Network Emulation with Linux Netem

Back in the day, I used an open source program called NistNET to emulate a WAN for my company’s network test lab on a linux machine.  I was able to solve a multitude of issues and test our applications in a WAN environment with this product.  Unfortunately, NistNET is no longer maintained, and until recently I had no open source tool for emulating a network in my arsenal.  The other day, while playing Call of Duty 2 with some friends on my dedicated linux server, I decided that I was tired of having an unfair advantage since my latency to the server was 1 ms, and they all had 50-70 ms or more, so I went on a search for something I could use to add delay to my connection to the server (one of my buddies says I am too honest).  After some searching, I came upon netem, which to my surprise is and has been part of the linux kernel for some time.  I know, some of you linux guys and gals out there are saying “tell me something I don’t know”  but, ashamedly, I didn’t know about this one perhaps because I don’t do much of that kind of work any more.

Anyway, using Netem, I could do exactly what I wanted.  I can add enough delay to my client to game server traffic (actually, its server to client, which I will explain later) to make it seem like I have a lot more network between the server and me than I actually do.  Whether or not this decreases my advantage in the game is yet to be seen.

Anyway, several “effects” are present on most wide area networks today.  A common effect, latency, can have a drastic effect on the way network communications protocols behave.  Latency is also one of the key issues with playing on line games, especially those that require fast reaction to on screen events.  High latency creates what gamers refer to as lag.

The Netem function of linux provides the capability to modify the parameters of egress traffic (ie, traffic exiting the machine and destined for another point on on the network).  With netem, its possible to create artificial delay, thus creating latency.  Other possibilities are rate limiting (controlling the amount of bandwidth traffic can use), causing packet loss, and jitter.  Packet loss can result in very poor performance with TCP applications.  Jitter, also known as variable delay, is bad for real time streaming applications such as voice over IP.

Anyway, you could probably see why this kind of stuff would be important to a network engineer, especially in a lab environment.

So – on to how to use netem.  Netem is controlled by the tc command, which is part of the iproute2 package and is included in with most linux distributions.

Using the tc command, we can easily tell a linux host to delay all packets exiting a network interface using this command:

tc qdisc add dev eth0 root netem delay 80ms

This will add 80ms of delay to all packets leaving the eth0 interface.  To test the result of this command, just do a ping from you machine before issuing the command, and then after:

ping -n

PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.103 ms
64 bytes from icmp_seq=2 ttl=64 time=0.394 ms
64 bytes from icmp_seq=3 ttl=64 time=0.070 ms
64 bytes from icmp_seq=4 ttl=64 time=0.101 ms
--- ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2998ms
rtt min/avg/max/mdev = 0.070/0.167/0.394/0.131 ms

Enter the tc command for adding delay to eth0:

tc qdisc add dev eth0 root netem delay 80ms

Then ping again:

ping -n
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=80.0 ms
64 bytes from icmp_seq=2 ttl=64 time=80.0 ms
64 bytes from icmp_seq=3 ttl=64 time=80.0 ms
64 bytes from icmp_seq=4 ttl=64 time=80.4 ms
--- ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 80.073/80.164/80.414/0.246 ms

Notice the difference in delay (~80ms).

We can also add variable delay (jitter) as most wide area networks (such as the internet) have some jitter associated with them.  The following command will add +/- 10ms of jitter to the 80ms delay shown in the last example:

tc qdisc add dev eth0 root netem delay 80ms 10ms

Now lets do the ping again:

PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=72.6 ms
64 bytes from icmp_seq=2 ttl=64 time=84.6 ms
64 bytes from icmp_seq=3 ttl=64 time=86.7 ms
64 bytes from icmp_seq=4 ttl=64 time=84.0 ms
--- ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 72.648/82.023/86.752/5.510 ms

Looks even more like a real internet connection now.

To see what qdisc (short for queuing discipline) parameters have been applied to an interface (in this case eth0) use the following command:

tc qdisc show dev eth0

Sample output follows:

qdisc netem 8003: root limit 1000 delay 80.0ms  10.0ms

The last part of the output shows that a delay of 80ms +/- 10ms is applied.

Now, for the important part – how do you turn this off? It took a while to find this in the netem documentation:

tc qdisc del dev eth0 root

This will remove all queuing discipline parameters from the eth0 interface on your system.

So this is great, but not necessarily what I am looking for.  Adding delay wholesale to the server would also increase my fellow gamers latency as well as mine, and the idea is to level the playing field.

That is ok, since netem/tc has a way to only place qdisc’s on specific traffic.  In my test network, I have two machines.  One running Windows 7 (this case the Call of Duty Client) and one running Ubuntu 9.10 (the COD2 server).  The Windows machine has an IP of, and the server

On the linux server, I run the following commands as root:

tc qdisc add dev eth0 root handle 1: prio
tc qdisc add dev eth0 parent 1:3 handle 30: netem delay 140ms 10ms distribution normal
tc filter add dev eth0 protocol ip parent 1:0 prio 3 u32 match ip dst flowid 1:3

This set of commands creates a simple priority queuing discipline, attaches a basic latency netem at hook 3, and then tells all traffic to to be priority 3, thus subject to the netem delay of 140ms +/- 10ms (with a normal statistical distribution of jitter).

These commands do exactly what I was wanting – making my delay to the game server about equal to my friends.

So far, it seems to work, however it’s not optimal.  The reason its not optimal is because only the packets coming from the server to my Windows client machine are being delayed.  A true internet connection would have delay in both directions.  Since netem only affects the egress of traffic from a network interface, technically you would have to delay the traffic as it leaves the client PC, and delay the traffic as it leaves the server back towards the client.   Since Windows doesn’t have a netem facility (at least not without some expensive commercial software such as that from Shunra) the best way to do this would be to run Call of Duty 2 on Linux using wine (which is another article for a another time).  That way I could induce delay on both machines, and get a “more perfect”  simulation of the internet.

To show existing filters such as those set by the last set of commands you can use the following commands:

tc filter show dev eth0


tc qdisc show dev eth0

Here is an example output:

tc filter show dev eth0

filter parent 1: protocol ip pref 3 u32
filter parent 1: protocol ip pref 3 u32 fh 800: ht divisor 1
filter parent 1: protocol ip pref 3 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:3
match c0a8000f/ffffffff at 16

tc qdisc show dev eth0

qdisc prio 1: root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc netem 30: parent 1:3 limit 1000 delay 140.0ms  10.0ms

In summary, netem is a perfect example of what I love most about Linux – flexibility and utility beyond what most commercial Operating Systems (especially those from Redmond) offer out of the box for free. While the way I have implemented it is not quite perfect, netem can provide what I am looking for in terms of simulating network conditions. As I mentioned above, perfection could be achieved by using linux as the client as well as the server, or by making a bridge between the two machines with a dual interface linux machine doing netem delay on both interfaces.

Netem has several capabilities that I didn’t cover, which can be found at the following links:

The Linux Foundation has an overview page here.
Here is pdf file showing more details on netem and tc usage.

Next up – Client IPSEC VPNs from a linux laptop to a Juniper Netscreen VPN/FIrewall device. Coming Soon!

SSH Tunnelling (aka Poor Man’s VPN)

Tunnelling of TCP traffic can be performed from the ssh command on Linux or with Putty on Windows, and can be thought of as a poor man’s VPN.  A VPN is a virtual private network, or a method of using a public network such as the internet to  securely transmit data via an encrypted “tunnel”.

VNC is a method of gaining access to a remote GUI on Linux and Windows machines.  VNC is typically considered insecure and not recommended for use on the open internet.  With an SSH tunnel, this doesn’t have to be an issue, as ssh provides security to an otherwise insecure protocol.

Here is an example of how to use VNC over an SSH tunnel:

Start VNC server on a Linux host ssh server to only listen to Loopback interface:

vncserver :1 -localhost

On client machine, start ssh with the following command line:

ssh -L 5901:localhost:5901 <server ip> [-l <login>]

The to access VNC via the SSH tunnel, use the following command on the client machine:

vncviewer localhost:1

What happens?

The ssh process on the client sets up a TCP port redirection on the loopback interface of port 5901, to the loopback interface on the server machine on TCP port 5901.  The vncviewer command connects to the the redirected port on the local loopback interface, which then gets directed over the tunnel to the server machine’s loopback on port 5901, where the vncserver is listening.

This will allow vnc protocol to be securely tunnelled across the SSH connection.

The previous examples showed connecting to services running on the ssh server itself.  Its also possible to use the SSH server to redirect traffic to other machines on the network behind it.

Sometimes we might have to access a Windows Machine behind an a linux SSH server that is connected to the internet (such as linux system performing firewalling for a home network). We can use SSH tunnelling to connect to Windows Remote Desktop as well. (I know this is a linux blog, but most of us out there still have to deal with Windows from time to time)

The following example assumes the following network layout:

Client PC –> Internet–>SSH Server on Firewall–>Private Network–> Windows XP
(                                                                                                        (
(ssh server can be behind firewall as long as its accessible from Internet)

1. Make sure Windows XP host is running RDP

2. On client PC, start SSH with tunnelling as follows: 4000:

On Putty this tunnel definition looks like this (click add after completing the boxes):

Which is exactly like the ssh command on Linux:

ssh -L 4000: <server> -l <userid>

To connect to the RDP service on the internal Windows XP system, from a client Windows system connected to the internet via the ssh connection:

Use Remote Desktop Connection application that comes with XP, but use this as the address to connect to:

Like the previous example, this causes the program to connect to port # 4000 on the local loopback interface, which then is redirected to the machine on port number 3389 at the other end of the ssh tunnel.  Port # 4000 is used to avoid conflicting with port #3389 on the client as it could have its own RDP server running.

Any TCP based communications can be tunnelled this way over ssh, creating a secure connection for any unsecure protocol. This is also a mechanism for bypassing firewall rules.  As long as the SSH server traffic is allowed (TCP port 22).  It can be used to gain access to other ports that might not be allowed by a local firewall, simply by using a remote ssh server as a proxy for other traffic.

Boot Sector Management

As promised, tonight we explore boot sector management on X86 style hardware.  Anyone who works with PC hardware long enough, and especially those using linux as primary or secondary OS in a dual boot configuration will find this information valuable.

The system boot sector on x86 style hardware is crucial to being able to boot a linux system on this common platform.  Occaisonally the boot sector becomes corrupted or needs to be backed up.  In the days of MS DOS systems, a command was used to “restore” the boot sector.  The command was


Essentially this would re-write the boot sector on the primary hard disk.

The dd command can be used to perform similar functions, however as is usual with Linux, more boot sector related tasks can be accomplished.

First of all lets review the structure of a boot sector or master boot record on a PC hard disk:

Format of the boot sector:

Size (bytes) Description
446 Executable code section
4 Optional Disk signature
2 Usually nulls
64 Partition table
2 MBR signature

The first 446 bytes of the boot sector contain executable code that is loaded by the BIOS and then executed, and is where OS boot loaders and boot managers (such as grub) store their initial code.  Its also an area of the disk that can become corrupted, or replaced during operating system installs.

The other part of the boot sector that is significant is the partition table.  This is where the disk partition information is stored.  This should not be modified by anything other than a disk partitioning utility such as fdisk.  It can be backed up for data security reasons though.  The total bytes in the master boot record comes to 512.  With dd, simply reading or writing the first 446 or 512 bytes of the disk device will read or write the master boot record.

Scenario 1:  Backup the boot sector (or MBR)

If the first harddisk in the system is /dev/sda, to backup the boot sector the following command can be used:

# dd if=/dev/sda of=bsbackup.bin bs=512 count=1

Essentially this command will read the first 512 bytes of /dev/sda and write it to the file bsbackup.bin.

Scenario 2: Restore the boot sector from a file:

# dd if=bsbackup.bin of=/dev/sda bs=512 count=1

This will restore the boot sector to /dev/sda that was backed up in Scenario 1.

Scenario 3:  Zero out the boot sector (leaving the partition table intact)

Sometimes a virus or other issue can leave a corrupted executable code section in the MBR.  I have personally seen a boot sector that would not store grub information (and thus boot linux after its installed) properly until the first 446 bytes were zeroed out and grub re-installed.  The following command will do just that:

# dd  if=/dev/zero of=/dev/sda bs=446 count=1

Scenario 4:  Zero out the entire MBR (this will erase the partition table as well – effectively destroying the ability to easily access data on the drive)

A variation of the last dd command will wipe out the master boot record entirely.  You will have to repartition and reformat your hard disk after this:

# dd if=/dev/zero of=/dev/sda bs=512 count=1

In summary, the use of dd for boot sector management is a handy tool to have in your linux arsenal.

Next up are some networking topics, such as SSH tunneling, IPSEC VPNs.  Keep watching the site, or subscribe to our RSS Feed.

Disk and Partition Imaging using dd

Linux provides an abundance of advanced command line tools to manage and modify just about anything on your system.  Today we will explore the use of dd, the primary tool on linux for creating and restoring disk images, among other things.

The dd (diskdump) on Linux can be used to backup an entire disk or partition to an image file. Several caveats apply to this method:

  1. The disk in question can not be in use by an operating system
  2. A destination medium or network resource must be present that is large enough to hold the image.

To backup a disk using dd, the following procedure can be used.

  1. Boot the computer with the disk in question from a Linux Live CD, such as Ubuntu or Knoppix
  2. Mount a destination disk (such as a usb disk drive or nfs mount)
  3. Run dd command to backup disk
  4. Note the size of the disk partition if partitioning a new device is necessary when restoring the image

Here is an example session, to back up a single partition (sda1) containing a Windows XP installation to a USB hard disk mounted at /mnt/sdb1:

As a root user, do the following:

Mount USB disk drive

# mount -t ext3 /dev/sdb1 /mnt/sdb1

Run dd command (piping output through gzip to save space):

# dd if=/dev/sda1 conv=sync,noerror bs=64k | gzip -c > /mnt/sdb1/windowsxp-c.img.gz

Definition of the dd command parameters:

“if=/dev/sda1” is the input file for the dd command, in this case, its linux device sda1
“conv=sync,noerror instructs dd that if it can’t read a block due to an error then it should at least write something to its output of the correct length.

Even if your Hard disk exhibits no errors, dd will read every single block, including any which the OS avoids because it has marked them as bad.

“bs=64k” is the block size of 64 kilobytes. Using a large block size speeds up the copy process. The output of this is then passed to gzip for compression and storage in a file on the destination device.

Noting Partition configuration:

Using the command fdisk -l /dev/<device> where <device> is the device node of the disk being backed up, make note of the number of blocks used to create the partition:

# fdisk -l /dev/sda

Disk /dev/sda: 959.9 GB, 959966085120 bytes
255 heads, 63 sectors/track, 116709 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x2e2d2e2d

Device Boot Start  End   Blocks     Id System
/dev/sda1*      1  15298 122881153+ 7  HPFS/NTFS
/dev/sda2   15299  51771 292969372+ 83 Linux
/dev/sda3   51772  52767   8000370  82 Linux swap / Solaris
/dev/sda4   52768 116709 513614115  83 Linux

The destination disk should have a partition defined identical to the source partition, total number of blocks is the important parameter here.

The partition geometry information can be backed up the the USB hard disk with the following command:

# fdisk -l /dev/sda > /mnt/sdb1/sda_fdisk.txt

Restoring a dd image to a disk/partition

The steps are similar to the backup process:

  1. Boot computer with destination disk from a Linux Live CD
  2. Partition the destination disk, if needed
  3. Mount the source media (usb disk or nfs mount)
  4. Use gunzip and dd to restore image to disk or partition

Here is an example session, to restore the image taken with the above steps:

As a root user, do the following:

Mount image source (USB hard disk at /dev/sdb1)

# mount -t ext3 /dev/sdb1 /mnt/sdb1

Partition destination disk:

# fdisk /dev/<device node in question>; in our case, sda.

<create partition if needed>

Restore image (destination partition is /dev/sda1)

# gunzip -c /mnt/sdb1/windowsxp-c.img.gz | dd of=/dev/sda1 conv=sync,noerror bs=64k

Note: On a fast machine, ie C2Q 6600, and 3ware RAID disk array, a 120GB image takes 25 minutes to create.

In order to have a bootable system, some other configuration may be needed such as restoring a boot block. Check out the next post for details on boot sector management with dd.

A excellent guide on using fdisk for disk partitioning can be found here.

Hello, world!

Welcome to!  With this blog, I intend to publish my experiences using Linux as a desktop and server operating system.  I have been using Linux since 1995 and around 2000 I completely replaced Windows with it as my primary personal computing environment.

I use linux for personal productivity, gaming, multimedia, programming, and as a server operating system, and I hope to share my knowledge of this powerful environment here on this blog.

A little about myself:

I am IT systems professional with 17 years of experience in design, maintenance and management of enterprise networks.  I am currently responsible for planning and strategy of Network infrastructure and Unified Communications technology investments for a major retail company based in the United States.

My favorite Linux Distros: Ubuntu, Gentoo.

Hobbies:  Linux, Data Communications, Computer Programming, Amateur Astronomy, Snow Skiing, Boating and Water Skiing, Biking, Computer Gaming, Beagles (I have three), Electric Guitar, Science Fiction Novels, Aviation

Favorite Music: Pink Floyd

Enjoy, hopefully you will find something valuable here if your interested in using Linux!  Stay tuned for more posts, as I have a bunch of content to share.