Beyond Syntax | Michael J. Schultz

On Demand ssh-add

2012-01-06T00:00:00-08:00

For many years I have been bothered by the options for using the ssh-agent on Linux. I finally (hope) I've reached a reasonable solution for adding my keys to the ssh-agent.

Background

As many *nix users know: SSH keys with passphrases are the way to go when jumping from system to system. This is the case because on your main computer there is a program running called ssh-agent. When you first boot your computer ssh-agent knows nothing and doesn't do much.

The real magic happens when you tell ssh-agent about your SSH keys. In the past I've done this in one of two ways:

Use the ssh-add command to add a key to the agent.
Have the desktop environment ask for your passphrase when it loads.

Since these have the same result of adding your passphrase to ssh-agent it simply becomes a matter of user experience. Neither experience I really enjoyed.

In the first case, my use pattern was always:

$ ssh <hostname>
Enter passphrase for key '<key>':

At which point I would curse, hit control-c, run ssh-add, and try sshing into the host. Not terrible, but not pleasant either.

The second option would pop up some graphical prompt asking for my passphrase when starting the desktop environment. This is also not a terrible experience, but it's a bit annoying when I'm just sitting down and don't want to type a long passphrase just to check my email. (I could close it, but then I'm back in the frustrating first case.)

What is a geek to do?

Honestly, the OS X way of doing this isn't too bad. If I haven't authenticated myself to the computer when I ssh the first time OS X recognizes that and prompts me for authentication. At this point I'm authenticated and any future SSH attempts will re-use that authentication. That is to say:

OS X automatically adds the authentication to the ssh agent*
I'm not bothered with authentication until I need it.

* This isn't quite how OS X actually does it, the passphrase is stored in a keychain. The keychain gets unlocked when I need it and serves the passphrase to the ssh command.

I want the same experience on Linux.

After searching for some tools to do it for me (and finding nothing), I decide to take a stab at it. It turns out to be very easy. All I did was add the following line to my ~/.bash_profile:

alias ssh="( ssh-add -l > /dev/null || ssh-add ) && ssh"

This just checks to see if any identities exist in the running ssh agent (ssh-add -l > /dev/null returns 1 if it is empty and 0 otherwise). If there are identities, the || operation is short circuited and the ssh command is run as normal. If the agent is empty, the ssh-add command runs and asks for your passphrase as normal, adds it to the agent, (returns 0), and runs the ssh command as normal. In both cases, any command line arguments are passed on to the final ssh command as normal.

If you mistype your passphrase and ssh-add gives up it returns 1, causing the && operation to short circuit and the normal ssh command is not run and you can try again.

Of course, this makes a few assumptions:

Either your agent is empty or it has all useful keys in it.
You're going to ssh before you scp or use any other command.

In my case both assumptions a valid. I don't have any keys that ssh-add doesn't find automatically (even if I did, I would still want to manually add them in them to the agent--in which case I am no worse off than before). In the second case, I almost always ssh to a host before moving files around or to, just so I can make sure I have the locations correct. (Of course, as I submit this post git asks for my passphrase because I haven't authenticated yet. The quick answer is to alias all commands that need ssh key authentication with the stuff before && and replace ssh with the desired command.)

This also shouldn't hurt security any more than another solution. At the end of the day, I'm disappointed that I didn't do this a long time ago.

Sophos Holidy Puzzle Walkthrough

2011-12-21T00:00:00-08:00

In case you missed the it there was a short holiday puzzle challenge over at the Sophos Naked Security blog. If you want to attempt solving it yourself I'd suggest going over and checking it out now before reading on because there will be plenty of spoilers. There is also a more Windows centric version on Belahzurs Thoughts, if that's what you would rather see that. Now, on with the challenges!

Step 1

The only hint is: =ImYndmbn1ieiBnLmJWdjJmZ.

The first things I noticed about this were the = at the beginning and the character set consisting of [A-Za-z0-9], meaning this is likely a reversed Base64 encoding. (Note that Base64 also includes + and /; they just don't appear here.) Using Python, these two actions are simple enough:

>>> '=ImYndmbn1ieiBnLmJWdjJmZ'[::-1].decode('base64')
'fbcubf.pbz-gnggbb'

That's still junk though, but now the junk looks a bit like a web address. Each letter is probably just shifted over a little bit (i.e. a becomes n). The most common form of character shift is to rotate the alphabet 13 characters ("rot13"). Well Python can do that as well:

>>> 'fbcubf.pbz-gnggbb'.decode('rot13')
u'sophos.com-tattoo'

(The preceeding u just means the string is in unicode.) Replace the - with a / and we've got the web address for step 2.

Step 2

Visiting the Step 2 site gives us our second puzzle: a text file with some hints and the following (suspicious block of text).

l504b0i304140st00hbs000800ld26492n3f0496707b4rb010000290400t0027001hcg0073656i3757r26974792d616476
6963652d6l66f722d74726w1i6t96e2d636f6d6d75746572732e67696655540900h031b44etdh4e3344ed4e75780b00010
4f5010000041r4000000051gafao6986186b0dded84f13cbe5f3dente45dc4e786tt03a1ob4775a0b6104o83df1c74498a
1l447f7ad0id==1cb7abd==c84f904=======..======f184=======0.===========.2==256c==d3446ffsb2d830825c0
d320ae6fd64|  |etde2|  |83cba/       ||   _  \3c|   ____||           ||  |4h|  |54ddbf6b47asdl2063
8n34r58be58|  |840d6|  |04th|   (----`|  |_)  |0|  |__30g`---|  |----`|  |==|  |iaderf142179334fld
bwci6cbdfdt|  |91b52|  |6225c\   \d2aa|   _  <63|   __|328dbe|  |3d1c7|   __   |hf87ta3698hr193340
de1f9gdf023|  `====.|  |1.----)   |014|  |_)  |3|  |____f0o87|  |e72en|  |14|  |4t8dct07cde1964dd7
tf914o05b3e|_______||__|f|_______/6a7b|______/c5|_______|95fb|__|85oef|__|al|__|84611d43ai44acsc50
0c154e8t7283b7fa2hsf3lc7bdnba80dca3a8c43r8t1aee241476424a9c52c8060579hdgi6r0414ablbc7wfa7ec3i27et6
5790800013681030ahb11466a84dth81rdge35c538b34d706697a867ff5df7a706156o36e97cadden1t326306t57343718
173t7e92=======.055ao===6371ed6==oea0f7698ea===26c862.==add==.ld=======3di=======a.======7s44493bf
d124179/       |tb70/   \88374|  |010a6h025/   \1af2a|  \s|  |l|       \e|   ____||   _  \5822c768
80c9b2|   (----`722/  ^  \18e9|  |bdf3n50c/  ^  \1bf3|   \|  |2|  .--.  ||  |__3dd|  |_)  |e26276r
f0d8dd9\   \bdt66f/  /_\  \4ch|  |fc295d2/  /_\  \23f|  . `  |9|  |gi|  ||   __|2f|      /d89059c0
e25.----)   |52r6/  _____  \34|  `====.b/  _____  \5l|  |\   |0|  '--'  ||  |____4|  |\  \====.b07
w08|_______/0496/__/707b4\__\b|_______|/__/01000\__\0|__|2\__|9|_______/0|_______||4_|0`._____|000
504b01021e0314000b0008i00d2t64923hf049t670h7b4b0100002904000r0g27001800000000000000000on0a48100000
000736563757269t74792d61t6t4766963652d666f722od7472616o96e2d636f6dl6d75746572732e6769665554050i003
1bs44ed4the75780bs000ln104f50100r000t4140000005h04b050600000g0i00010001006d000000bc01000000r00lwit

This is most likely the starting point to the puzzle and one of the hints was that we'll need a password in this stage of the puzzle. After a bit of thinking I figured that a Zip file is the most likely candidate for cross-platform password protection, but how is that block of text a Zip file? Looking through the Wikipedia page shows that PK is the commonly the first two bytes of a Zip file, it just so happens that that translates to 504b in hexadecimal. Perhaps if I strip all the non-hex characters from the text block it will be a Zip file. This time I'll just use the Unix tr utility to strip non-hex characters:

$ tr -cd [A-Fa-f0-9] < topsecret.in > topsecret.hex

This deletes the compliment of the character set [A-Fa-f0-9] from the input file topsecret.in and outputs to topsecret.hex. The next step is to convert the ASCII representation of the hex digits to binary. Again, I turned to Python for this:

>>> infile = open('topsecret.hex')       # Sets up a file descriptor
>>> line = infile.readline()             # Reads the first line from infile (the only line)
>>> b = ''                               # An empty string we'll build on
>>> for c in range(0, len(line), 2) :    # Go through the characters (in steps of 2)
...     v = int(line[c:c+2], 16)         # Convert 2 ASCII values into their actual value
...     b += chr(v)                      # Append the actual value to b string
...
>>> infile.close()                       # Close the input file
>>> outfile = open('topsecret.zip', 'w') # Open the output file in write mode
>>> outfile.write(b)                     # Write the contents of b to outfile
>>> outfile.close()                      # Close the output file

Now we've got a Zip file ready to go! We know the password was the cipher used to get from step 1 to step 2, so just unzip the file:

$ unzip topsecret.zip
[...] password: rot13
  inflating: security-advice-for-train-commuters.gif

Open up the file and you get to see a fantastic shade of pink!

We are probably not done quite yet, eh? We can look at the actual contents of the file by using xxd:

$ xxd security-advice-for-train-commuters.gif > security-advice-for-train-commuters.gif.dump

Opening that dump file up in a text editor and we see "Since when was pink a shade of gray?" Hmmm, maybe if we convert part of the from pink to gray we'll see something new. From years in web development you may know that #f1bbed is a shade of pink and that value shows up twice in the GIF file. I decided to change the first instance to 444444 a nice dark shade of gray. Now we can reverse the hex dump output back to the binary GIF format:

$ xxd -r security-advice-for-train-commuters.gif.dump > final.gif

Open that file and you get a much more clear version of the clue.

Probably not the final answer (it's an anagram). (I knew it was an anagram, but I couldn't figure out what it was saying so I emailed Paul Ducklin for a hint. -- I got stuck thinking "security" was in it with the "?" being any character.)

The hint was in a recent article about Lost USB keys, with the clue being "what is the article trying to tell you?"

Taking out "USB" from the image, it isn't too hard to determine the final answer:

ENCRYPT YOUR USBS

I'm sure there are better tools to get to the answer, but this is how I did it!

Diving into the Linux Networking Stack, Part I

2011-03-13T00:00:00-08:00

This is the first part of a (planned) multi-part series on receiving a network packet on a modern Linux kernel.

Have you ever wondered what happens when your Linux machine gets a packet from the network? I have, but most of the information I've seen is concerned with the 2.4.x kernel. For my own sake, I decided to take a walk through the Linux networking stack (using Linux kernel 2.6.37) and thought someone else might be interested in a little breakdown as well.

Starting at the Driver

I've decided to take a bottom up approach and begin with software that interacts with the physical network card, the driver.

In general network drivers follow a fairly typical route in processing: the kernel boots up, initializes data structures, sets up some interrupt routines, and tells the network card where to put packets when they are received. When a packet is actually received, the card signals the kernel causing it to do some processing and then cleans up some resources. I'll talk about the fairly generic routines that network devices share in common and then move to a concrete example with the igb driver.

A General View of Network Drivers

Network drivers are far from the simplest drivers in the kernel. They push the speed boundaries of modern multi-core processors. Almost every new system comes with a 1 Gigabit per second (Gbps) network card. Taking into account that the smallest Ethernet frame size is 64 bytes (plus 8 bytes for synchronization and a 12 byte inter-frame gap), a 1 Gbps network card should (by specification) be able to handle:

This leaves us with a processing time of about 672 nanoseconds per frame. A 2 Gigahertz processor can execute about 1,400 cycles in that time and a single instruction can take multiple cycles.

Is a modern operating system able to keep up with that rate? The simple answer is: "no." The standard Linux kernel can't keep that pace for an extended period of time it relies on buffering to handle short bursts of traffic beyond what it can handle. It still does its best to get as many packets as possible using modern network card resources and the multi-core processors available on the machine.

A Quick Refresher on Driver Basics

During the boot process the kernel discovers the network card and sets up its data structures. These data structures include multiple queues for receiving packets while the kernel trying to process them. Each receive queue is independent from the rest so a processor core can work through its backlog safely. To prevent wasteful context switching, interrupts are disabled while a core works through the packet backlog. The network card also writes packets directly to memory so no processor resources are used to move the packet from one place to another.

How does this all happen so quickly? It all depends on some careful coordination between the kernel and the hardware. Since every piece of hardware is different the kernel creates a standard interface between itself (or the user) and the underlying hardware through a "driver." The driver provides the abstract kernel functions by implementing them for the actual hardware. The driver uses whatever tricks available to fufill a request. Network drivers typically give the hardware a region of memory, that describes separate memory locations it can write packets to, and carefully allocates/re-allocates resources as needed.

In the figure above, you can see the main components involved in the low level packet reception: main memory, the network card, and the Linux kernel.

On the left side, the network card holds a descriptor ring. An entry in the descriptor ring points to a location in main memory (which was set up to be a socket buffer) where it will write the packet. Entries can also contain information about the packet or the state of the network card during reception.

On the right side, the Linux kernel maintains a pool of socket buffers. The pool concept works well here because the network card needs quick access to socket buffers, but once used a socket buffer can remain active for a while so the network card can go back to the pool and grab a new one. There can also be a simple wrapper between the socket buffers and the descriptor ring to decrease overhead when moving around the network stack.

Finally, the only real direct communication from the network card to the kernel happens by way of an interrupt. Once the network card finishes receiving a packet it signals the kernel, by way of an interrupt, that it has processing to do.

A Concrete Example in `igb`

I'm going to work through the Intel Gigabit Ethernet (drivers/net/igb/) driver. All functions that I'll discuss here are in igb_main.c, unless otherwise noted. The other files in that directory are very hardware specific functions that lets the software communicate with the hardware, the implementation of these is usually just a matter of reading the hardware specification and writing software that follows the specification.

Initialization

Typically during the boot process the driver is loaded into the kernel as a kernel module. When the module is registered with the kernel it executes a callback function called igb_init_module. This function registers the driver with the PCI bus and provides another callback which is executed once the PCI bus is configured. Once the PCI bus is ready, we end up at the igb_probe function.

In igb_probe the driver actually enables the device on the PCI bus, gets memory for PCI device input/output, sets device specific callback functions (like open and close), calls igb_sw_init (which sets some software state and prepares the interrupt system), configures some hardware details, and ensures the device in a known state. Each step is important, but there isn't anything particular to networking devices because this is generally what any PCI device has to do before it is ready. The real networking "meat" happens when the device is opened through the igb_open callback that was registered in this stage.

Opening the Device

When the system actually brings the device up, it calls the open function which n this case is igb_open. This is where the heavy lifting starts happening. All the resources needed to send and receive packets are allocated and the interrupt handler is set up so Linux knows what to do when a packet is received. Since I'm more interested in packet reception than transmission, I'll focus on those parts.

Allocating Resources

First, we allocate the transmit resources (igb_setup_all_tx_resources) then the receive resources (igb_setup_all_rx_resources). Modern network are able to use multiple queues (rx_rings)---called "Receive Side Scaling" (RSS) on Intel cards---allowing them to distribute the load amongst processors. Thus, the process of setting up all the receive resources will allocate the resources for each queue.

The resources in this case is a wrapper buffer (struct igb_buffer in drivers/net/igb/igb.h), which acts as a software link between the hardware specific descriptor ring and the software oriented socket buffer. At this point, none of the socket buffers have yet been allocated but the hardware knows about the descriptors which will tell it where to place packets when they start arriving.

After the wrapper buffers have been allocated and the hardware is given the descriptor ring we must associate each wrapper buffer with a socket buffer and update the descriptor to point to the socket buffer location. This happens in the igb_alloc_rx_buffers_adv function (which is called from igb_configure). If the wrapper is not associated with a free socket buffer (struct sk_buff in include/linux/skbuff.h), it will allocate a new one and set the hardware descriptor to point to the correct locations. The socket buffer is mapped between hardware and software via direct memory access (DMA) so no memory copying must be done as it moves through the network stack.

Now that the wrapper is associated with both a hardware descriptor and a software socket buffer we are almost ready to receive packets.

Configuring Interrupts

Now that all the memory resources are allocated the only major hurdle remaining is setting up interrupts. There are two interrupt routines that the driver must set up: the hardware interrupt routine and the software interrupt routine.

From the igb_open function, the igb_request_irq function is called. Again, I'll make the assumption that modern systems are capable of using the multiple receive queues and therefore will also take advantage of the Message Signaled Interrupts (MSI-X) which allows more interrupts than there are physical pins for interrupts. With a unique interrupt for each receive queue, the kernel is able to know exactly what caused the interrupt and a user can pin interrupts from a specific queue to a specific processor core to minimize detrimental cache effects. Combining these fact, each receive queue on the network device requests its own interrupt number and sets the hardware interrupt routine to be the igb_msix_ring function.

At this point when a packet comes in it'll hit the hardware interrupt, but once the hardware interrupt is finished there isn't a way for the higher level software to know about the packet. Enter the software interrupt function (softIRQ). Hardware interrupt handlers are designed to be fast, once done an associated software interrupt is informed and will execute in a safer context that won't block other interrupts. So the next thing that the igb_request_irq function does is register a softIRQ with the New Application Programming Interface (NAPI) layer via netif_napi_add (in net/core/dev.c function which adds a callback to igb_poll. The purpose of the NAPI layer is to mitigate interrupt overheads ("interrupt storms" or "receive livelock") under heavy packet load.

After a few more simple tidbits to make sure the hardware and software state matches, interrupts are enabled and the device is ready to send and receive packets!

Receiving a Packet

When a packet first arrives at the network card, the device looks at the next hardware descriptor and begins writing the packet to memory where the descriptor tells it to (this will be the location of the socket buffer data as configured during initialization). Of course, if all the descriptors the hardware knows about have been consumed it drops the packet, citing an overrun error. Once the packet has been fully received, the hardware asserts the interrupt signal and the kernel begins running through the associated interrupt routine (igb_msix_ring). This function is only four lines of actual code! However one of those lines is to napi_schedule (in include/linux/netdevice.h) which does the important stuff.

A hardware interrupt should be quick so the system isn't held up in interrupt handling. napi_schedule goes through a series of calls to __napi_schedule and ____napi_schedule (both in net/core/dev.c) that ends with the receive queue being associated with a specific processor on the system through a list and then signalling the software interrupt so it will run the next chance it gets. With the kernel now aware that a packet is available for processing on the receive queue the hardware interrupt is done, the hardware signal is un-asserted, and everything is ready for the next stage of packet processing.

Conclusions

I have now presented both an abstract and concrete view of lower-level network processing in a modern Linux kernel. We've gone from having no network card enabled, allocated the resources needed to receive packets in a multi-queue configuration, told the hardware about those resources through the hardware descriptors, configured the interrupt handlers, received a packet that was stored in memory, and told the kernel about the packet for further processing.

Now that the kernel knows a packet is on the receive queue, it will schedule the softirq handler which will call the igb_poll function, reconfigure the descriptors a little, and start performing higher level networking functions. The NAPI will come into play stronger than ever once the softirq is executed, but I'm going to save that for a future post.

Performance Monitoring with OProfile

2010-07-01T00:00:00-07:00

oprofile is a low overhead, open-source tool that hooks into Linux and can keep track of CPU event monitoring information. This is a fairly general statement and for this post I'll be using the Intel Penryn microarchitecture, which should have similar event counters to most recent Intel processors. You can get the canonical list of event counters from Intel's own documentation in Chapter 30, Performance Monitoring, of Volume 3B, System Programming Guide (available from Intel's site). Alternatively, the Japan Advanced Institute of Science and Technology have an interactive version with all the events for most Intel processors.

Event Counters

If you are unaware, almost every processor manufactured in recent history has some collection of event counters that are incremented when some processor event occurs. These events can range from clock cycles ticking by, instructions being retired, thermal thresholds being passed, or second level cache misses.

So far, I've only really used the CPU clock cycles, level 1 cache line replacement, and instructions retired event counters. Your needs might not match mine, so venture over to the Programmer Manual when you need something else!

Event Ratios

Related to the event counters are event ratios. These simple ratios can help you find specific performance issues in your program. For example, if your program does a lot of memory accesses, the processor may need to replaced cache lines frequently. But cache line replacements are naturally occurring in programs, how do we find excessive? Simple! We can just use the ratio of L1 cache replacements to the number of instructions retired. Then we'll have an idea of how many times per instruction an L1 cache line is replaced.

Using `oprofile`

First, you'll have to be running Linux, then you'll want to install the "oprofile" package. Since this software installs kernel modules for monitoring, you'll also need root/sudo access to allow the module to be loaded and unloaded for monitoring sessions. Here, I'll be running as a user and using the sudo command when needed.

`opcontrol`

opcontrol is main program that lets you interact with the kernel. If you need a down-and-dirty list of the events available for monitoring, opcontrol --list-events will show you all the event counters at your disposal.

On my processor, the default event to monitor is CPU_CLK_UNHALTED which will tell me where the processor spent most of the time executing. If you want to monitor different events, you can specify what event(s) to monitor at the command line. The separate flag simply tells the profiler to separate traces for each individual CPU (there are other separation levels, but we won't use them here).

$ sudo opcontrol --event L1D_REPL:10000 --event INST_RETIRED:10000 --separate=cpu

The :10000 after each counter simply specifies what the trigger threshold is for raising processor exception. In other words, every 10,000 instructions retired the processor raises an exception that the oprofile daemon will catch and then increment the sample counter for that event. So, if you see that oprofile has 1 sample of the INST_RETIRED counter then the processor has seen 10,000 such events.

Now that we have the event counters configured, we can start the monitoring.

$ sudo opcontrol --start

Since the system is doing other activities, it is best if what you want to monitor can monopolize the system for the while. In my case I build a simple program that purposefully causes the L1 cache to have a lot of misses (l1thrash source code). I'll also set the program to execute on one processor (CPU 1).

$ taskset 02 ./l1thrash

After it finishes executing, stop oprofile from running and save the profile session on the disk.

$ sudo opcontrol --stop
$ sudo opcontrol --save l1thrash

Now we have our profile saved to disk and we can view it with opreport.

`opreport`

Finally, we get to see how the program handled! Since we were smart and saved our profile to a session, we'll have to specify that at the command line. You might want to pipe the output to less since it can be long at times. On my eight core system the output looks ugly.

$ opreport session:l1thrash
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted L1D_REPL events (Cache lines allocated in the L1 data cache) with a unit mask of 0x0f (No unit mask) count 10000
Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7
    cpu:0|            cpu:1|            cpu:2|            cpu:3|            cpu:4|            cpu:5|            cpu:6|            cpu:7|
  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|  samples|      %|
------------------------------------------------------------------------------------------------------------------------------------------------
      541 95.7522      2969  0.9630       301 92.6154       484 69.9422       797 92.6744       707 88.2647       675 90.3614       707 89.8348 vmlinux
        7  1.2389        21  0.0068         6  1.8462         6  0.8671         9  1.0465         6  0.7491         6  0.8032         5  0.6353 oprofile
        6  1.0619         3 9.7e-04         6  1.8462         1  0.1445         3  0.3488         2  0.2497         1  0.1339         4  0.5083 nf_ses_watch
        5  0.8850        16  0.0052         6  1.8462         7  1.0116        25  2.9070        23  2.8714        30  4.0161        27  3.4307 libc-2.5.so
        3  0.5310         2 6.5e-04         1  0.3077         1  0.1445         2  0.2326         4  0.4994         0       0         1  0.1271 libpython2.4.so.1.0
        1  0.1770         0       0         0       0         0       0         0       0         0       0         0       0         0       0 e1000e
        1  0.1770         0       0         0       0         0       0         0       0         0       0         0       0         0       0 irqbalance
        1  0.1770         1 3.2e-04         1  0.3077         0       0         0       0         0       0         0       0         0       0 sshd
        0       0         2 6.5e-04         0       0         0       0         6  0.6977         8  0.9988         8  1.0710        18  2.2872 bash
        0       0         0       0         0       0         0       0         1  0.1163         0       0         0       0         1  0.1271 gawk
        0       0         0       0         0       0         3  0.4335         1  0.1163         2  0.2497         1  0.1339         0       0 bnx2
        0       0         0       0         3  0.9231         0       0         0       0         0       0         0       0         0       0 ehci_hcd
        0       0    305283 99.0179         0       0         0       0         1  0.1163         4  0.4994         1  0.1339         2  0.2541 l1thrash
        0       0        10  0.0032         0       0         0       0        14  1.6279        12  1.4981        19  2.5435        13  1.6518 ld-2.5.so
        0       0         3 9.7e-04         0       0         0       0         1  0.1163         1  0.1248         2  0.2677         2  0.2541 libcrypto.so.0.9.8b
        0       0         0       0         1  0.3077         0       0         0       0         0       0         0       0         0       0 libm-2.5.so
        0       0         0       0         0       0         0       0         0       0         0       0         1  0.1339         0       0 libpthread-2.5.so
        0       0         0       0         0       0         0       0         0       0         0       0         1  0.1339         0       0 syslogd
        0       0         0       0         0       0         0       0         0       0         1  0.1248         0       0         0       0 which
        0       0         0       0         0       0         1  0.1445         0       0         0       0         0       0         0       0 libcups.so.2
        0       0         0       0         0       0         0       0         0       0         0       0         2  0.2677         0       0 libusb-0.1.so.4.4.4
        0       0         0       0         0       0       189 27.3121         0       0        30  3.7453         0       0         7  0.8895 oprofiled
        0       0         1 3.2e-04         0       0         0       0         0       0         1  0.1248         0       0         0       0 cupsd

You may notice that the columns try to be sorted in descending order by the number of samples taken for a specific process. However, on CPU 1 (where we ran l1thrash) the sorted order isn't close to correct. Luckily, we know that the bulk of our program only ran on CPU 1, so we can reissue the opreport command specifying that we only care about that processor.

$ opreport session:l1thrash cpu:1
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted INST_RETIRED.ANY_P events (number of instructions retired) with a unit mask of 0x00 (No unit mask) count 10000
Counted L1D_REPL events (Cache lines allocated in the L1 data cache) with a unit mask of 0x0f (No unit mask) count 10000
INST_RETIRED:1...|   L1D_REPL:10000|
  samples|      %|  samples|      %|
------------------------------------
  1834500 91.0882    305283 99.0179 l1thrash
   154499  7.6713      2969  0.9630 vmlinux
    21655  1.0752        21  0.0068 oprofile
     2176  0.1080        16  0.0052 libc-2.5.so
      442  0.0219        10  0.0032 ld-2.5.so
      435  0.0216         2 6.5e-04 bash
      108  0.0054         3 9.7e-04 libcrypto.so.0.9.8b
       47  0.0023         3 9.7e-04 nf_ses_watch
       43  0.0021         1 3.2e-04 sshd
       35  0.0017         2 6.5e-04 libpython2.4.so.1.0
       10 5.0e-04         0       0 libavahi-common.so.3.4.3
       10 5.0e-04         1 3.2e-04 cupsd
        9 4.5e-04         0       0 libcups.so.2
        7 3.5e-04         0       0 bnx2
        3 1.5e-04         0       0 libavahi-core.so.4.0.5
        1 5.0e-05         0       0 libpthread-2.5.so
        1 5.0e-05         0       0 timemodule.so

That looks better! Since we've narrowed down the output to one CPU, we now get to see both events that we monitored too. You can see that the majority of the time was spent in our l1thrash program, but how did it do?

We know that the number of samples is the number of times that the event counter on the processor hit 10,000 for both counters. So, we find that our l1thrash program caused (305,283)(10,000) = 3,052,830,000 level 1 cache replacements and retired (1,834,500)(10,000) = 18,345,000,000 instructions. Egads! Is that good or bad? Well, now we can throw in our ratio calculation for the L1 data cache miss:

That seems pretty bad to me! We can also see that the Linux kernel (vmlinux) had a ratio of 2,969:154,499 or about 1.9%, that is a fairly typical miss ratio.

A Second Example

This is a real example of a program I am actively trying to improve. The program is a kernel module (nf_ses_watch) designed to intercept packets at a decent rate, it is not performing well. Here I'll use the default CPU_CLK_UNHALTED event monitor to see where the processor spends most of its time.

$ # I've already loaded the kernel module and started my packet generator
$ sudo opcontrol --event default
$ sudo opcontrol --start
$ # I'll wait about 30 seconds so there are enough samples to be meaningful
$ sudo opcontrol --stop
$ sudo opcontrol --save bombard

Now I have my saved session and can look at the profile. I've also taken the time to set the interrupt affinity of the Ethernet device to a specific processor (CPU 7), so now we can see if all the time was spent in my code of Linux code.

$ opreport session:bombard cpu:7
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
   737746 86.6169 nf_ses_watch
    88183 10.3533 vmlinux
    16810  1.9736 e1000e
     3680  0.4321 oprofiled
     2594  0.3046 oprofile
     1578  0.1853 libc-2.5.so
      900  0.1057 bash
       78  0.0092 ld-2.5.so
       52  0.0061 ophelp
       26  0.0031 libavahi-common.so.3.4.3
       22  0.0026 libavahi-core.so.4.0.5
       13  0.0015 gawk
        9  0.0011 libcrypto.so.0.9.8b
        9  0.0011 libpython2.4.so.1.0
        9  0.0011 sshd
        8 9.4e-04 bnx2
        4 4.7e-04 libpthread-2.5.so
        3 3.5e-04 grep
        2 2.3e-04 ipv6
        2 2.3e-04 auditd
        1 1.2e-04 cat
        1 1.2e-04 libdl-2.5.so
        1 1.2e-04 libm-2.5.so
        1 1.2e-04 libpcre.so.0.0.1
        1 1.2e-04 dirname
        1 1.2e-04 automount

Wow! Over 86% of the time we were executing code in the nf_ses_watch kernel module (my code)! Let's see if we can dig a little deeper. First, oprofile has already done the work for us and tracks the specific symbol name within a piece of code that was active when the sample was taken with the --symbols option (this results in a very long list of kernel symbols). But, in the case of a kernel module, opreport doesn't know where to find the symbol names so we have to tell it where the kernel module lives with --image-path.

$ opreport session:bombard cpu:7 --symbols --image-path ~/nf_ses_watch/kmod | head
warning: /bnx2 could not be found.
warning: /e1000e could not be found.
warning: /ipv6 could not be found.
warning: /oprofile could not be found.
warning: /sbin/auditd could not be read.
CPU: Core 2, speed 2494.04 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
warning: could not check that the binary file /home/mjschultz/mon/module/kmod/nf_ses_watch.ko has not been modified since the profile was taken. Results may be inaccurate.
samples  %        image name               app name                 symbol name
733996   86.1767  nf_ses_watch.ko          nf_ses_watch             do_rip_entry
16810     1.9736  e1000e                   e1000e                   (no symbols)
10308     1.2102  vmlinux                  vmlinux                  rb_get_reader_page
9785      1.1488  vmlinux                  vmlinux                  read_hpet
8701      1.0216  vmlinux                  vmlinux                  ring_buffer_consume
3606      0.4234  vmlinux                  vmlinux                  netif_receive_skb
3530      0.4144  vmlinux                  vmlinux                  kfree

(I've piped the output through head to keep it reasonable.) We can see the real dirt here! By a huge margin, the do_rip_entry symbol in my nf_ses_watch module executes more than the Ethernet driver that is handling the raw packets. So that is where I'll be looking when I try to resolve my bug.

Conclusions

If you are looking to optimize your program, oprofile is a great tool to use. The default event monitor (CPU clock cycles on most processors), can give you an idea of what part of your program is using most of the processor time. Once you know that, you can focus your efforts on reducing the number of cycles spent in that function. But don't forget about all those other events too. If you have a memory intensive application, maybe you could reduce the memory contention and get an effective speedup with almost no refactoring!

(I've tried my best to be accurate with this information and I welcome any explicit corrections or clarifications.)

The nth Backup Solution

2010-02-19T00:00:00-08:00

In the past, I had developed my own backup solution. Unfortunately, over time it didn't work out (mainly from changing systems, moving, using a laptop instead of a desktop, and maintaining it). However, I still like the idea of incremental backups as well as a mirrored version of my files (it saves space and lets me keep a history going back some number of days).

Now that I'm somewhat settled (and a little wiser), I decided to once more try my hand at a solid backup plan. This was mainly motivated by a recent reinstall of my wife's system (no lost data, just operating system upgrade). Since I don't have vast amounts of time on my hands, I didn't want to forward port my old solution to get it to work on Linux and Mac OS X, so I looked for new solutions. I recalled my advisor from Marquette mentioning rdiff-backup as what he put on his wife's machine during her dissertation days.

As it turns out, rdiff-backup does most of what I wanted out of my backup solution and, in fact, does it a little better. The main issue I had with my system was that it would periodically (monthly) take a snapshot of my home directory, after that it would periodically (weekly) build incremental diffs based off that snapshot. What this boils down to is that, if a catastrophic failure happens I would roll back to the most recent snapshot, then progress forward in time to the most recent incremental file. Not bad, but if you want better-than-weekly granularity it could be a lot of work. Obviously, I had scripted this part, but still it is wasted time. With rdiff-backup, it would be a single copy operation to restore to the most recent version. If you wanted older versions you could roll back through the incremental diffs (again, it is automated).

The other feature that I needed was the ability to remove backups/incremental data older than some time frame (monthly). Again, rdiff-backup gives me this ability at the command line. Other bonuses include the fact that it is cross-platform (via macports or most Linux repositories), written in Python, and not maintained by me!

With the basic service in place, it was time to make it automated. Again, linked off rdiff-backup's page is an article on how to do unattended backups. Besides the typical unattended SSH-keypair-without-a-passphrase and protecting-the-account steps, it introduced me to a new trick (which for some reason, despite having the knowledge on how to do it, never put together) using SSH config.

Host athena-backup
    Hostname athena.olympus
    User backups
    IdentityFile ~/.ssh/backups_rsa
    Compression yes
    Protocol 2

Now, if I try to ssh athena-backup, it'll automatically use the correct identity file and user name which saves me from having to specify it on the command line (which you can't typically do with wrapper functionality). More importantly, it doesn't break normal SSHing onto that host since we made it a special host (that's the part I never put together, even though I knew it was possible).

The next issue I never took that time to think about before was my having moved from desktop to laptop (thereby making 1:00am backups worthless sense the laptop isn't always on). Because rdiff-backup does a roll-back model instead of my roll-forward model, I decided to do hourly backups to my home machine, thus I'll likely catch at least one of these a day. But I'm not always at home! Getting around that is trivial, I'll just ping the backup server before trying. If it doesn't respond, I don't backup. This is done through:

$ ping -c1 -t1 $SERVER > /dev/null 2>&1

where $SERVER is just the name of the backup server. It pings the host once with a timeout of 1 second, if it succeeds the backup continues; otherwise the script exits.

Of course, setting up the cronjob is as simple as:

0 */1 * * * $HOME/.crontab/rdiff-backup.sh

Where $HOME/.crontab/rdiff-backup.sh is the path to my rdiff-backup shell script.

Hopefully this time around the backup solution is more robust than before.

Mozilla Fun

2010-01-22T00:00:00-08:00

I was just looking at some XML, and saw that the namespace for XUL is

http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul

It looks like someone likes Ghostbusters at Mozilla. I just found it amusing.

Technology and Courage

2009-10-30T00:00:00-07:00

A few weeks ago, Ivan Sutherland came to Washington University to give a talk to drum up interest in a new idea he is working on (Fleet, Infinity & Marina [PDF slideshow]). In my experience, most "old guy" talks aren't that interesting because they meander with long tangential stories about their children. Luckily, those were kept to a minimum and he had a good sense of humor too!

Now---interesting as the talk was---he suggested everyone read his only non-technical paper titled, "Technology and Courage" [PDF from Sun]. It took me until yesterday to read it, but it was certainly an interesting article. I recommend everyone read it.

RSS Sucks.

2009-09-29T00:00:00-07:00

I'll admit that I haven't spent too much time working with RSS feeds, but so far I'm unimpressed. All they really seem to provide is a consistent view of published data for clients to read when they want. That seems okay, but inefficient and a little redundant. It seems like you could implement the same thing by just sending an email to people who want to subscribe. At least then the end-user doesn't have to use both an email client and feed reader (yes, I understand some programs combine the two technologies). Alright, fine maybe you don't want to give the "evil-faceless-corporate-giant" your email address, after all you know they're going to sell it to someone. Is there a better way to publish data?

So, I'll start with what possessed me to write this. I'm trying to watch a Google Code project and I want to get updates whenever something changes. The "easiest" way to do that is through the RSS feed. But, I don't really want to download and use another application just to watch the feed for updates. Even if I did download the program, it wouldn't really gain me anything since it is just going to query ("poll") the server for new updates periodically, just like my email client already does.

I begin searching online for something that will watch RSS feeds on my behalf and send me an email when it updates. The first thing I come across is Feed My Inbox, they seem to offer the exact service I want. Upon closer inspection, they promise to only send one email every 24 hours. That won't cut it---I want my updates and I want them now! After a bit more searching I find rss2email, a simple Python program that keeps track of multiple RSS feeds and converts entries into emails when it executes. I go through the initial configuration and set up a cronjob to check for new entries every 4 minutes. Good enough for now.

However, this brings up an annoyance with RSS feeds:

RSS feeds do not provide real-time updates

Once you get down to it, all an RSS feed does is provide some subset of content on a page. It is still up to the client to ask the server when new content exists. This fact has bugged me very slightly in the past since I know xkcd updates every Monday, Wednesday, and Friday at 11pm central time, but my RSS subscription in Firefox won't provide me with the link until (at its discretion) polls the server for new content. Now, it bothers me slightly more since I know I have to wait at least 4 minutes for my cronjob to run, plus any time it might take Google to publish the updates in the feed (but I'll ignore that part).

Is there any way to improve this and give end-users real-time updates from content-providers? It seems like this would be great for users of Twitter and Facebook, since the end-users want to know what is happening now.

One answer seems to be in push notifications, that seem to have been popularized by Blackberry and iPhone applications. These allow a central server to send a tiny message to the phone that nudges the device that there is data to be had. Of course, this works very well on phone systems that can easily associate an content generator with a telephone number to contact. However, it is a bit tougher with IP-only devices that migrate from network to network. Although, it seems Apple's Push Notification Service (APNS) should be able to do this. Either way, this technology seems to be heading in the right direction.

APNS works by maintaining a connection between the client and server, that way when an event happens server side it just sends it on to all the connected clients. Unfortunately, I'm not convinced at how well these push notifications will scale. A system implementing IP push notifications seems like it could easily have on the order of 1000s of simultaneous, persistent connections. According to this "Comparison of Push and Pull Techniques for AJAX," (tech report, PDF) push-style system do bog down servers a bit. (Admittedly, the methodology for that paper might not be the best, but I'm guessing the conclusions are valid---I would like to see more/better studies of this.)

This bring me to what I want to see implemented or for someone to point me to the implementation of a distributed content syndication protocol (DCSP). The high-level view that I think would work (I haven't thought long or carefully about it), would be similar to other distributed networks. The content provider would maintain a complete list of current computers subscribed to the feed and the feed itself. The client would run software that asks the server who to connect to and select a few peers and create a long running connection. When new content arrives, the server pushes the content to its peers, who push to their peers, and so forth. This removes the burden of pushing content to all subscribers from the server, giving it scalability (in my mind). It would then be up to the client to connect and maintain connections with a collection of peers to get the real-time updates. I'm sure there would have to be some control messages to prevent flooding. But, it seems like it would give real-time updates to users.

I suppose this would mean I would have to run another program on my system, but it could either be a front-end client that handles my content feeds or a daemon running in the background and set up to deliver an email to a local mailbox (or even a remote mailbox) when fresh content arrives.

Ah well, who knows if it would work. Hell, maybe I just missed a fact about RSS that doesn't make it suck as much as I think it does. Thus ends my stream of though.

Remote Instance of Firefox via SSH -X

2009-07-27T00:00:00-07:00

Firefox is a pretty decent web browser. However, it can be a bit more clever than I want it at times. For example, if I want to SSH into a remote machine and launch a instance of Firefox -- to take on the remote machine's IP address or access localhost -- I would have to close the local instance then launch the remote instance. That is annoying and unacceptable behaviour.

Luckily, the solution is fairly straightforward. Once you have SSH'd into a remote host (using ssh -X), you simply need to run firefox -no-remote. Of course you may want to tack on > /dev/null and an ampersand & to ignore the output and background the task. (Thanks to The Open Sourcer.)

With Firefox 2.x this behaviour was somewhat undocumented, but with Firefox 3.x, running firefox --help from the command line shows the -no-remote option. It also seems that the default (i.e. -remote), is "documented" on Mozilla's site for RemoteControl of UNIX Mozilla.

If you wanted to make the -no-remote behaviour the default when SSH'd into remote machines, you could simply add a few lines to your bash profile to alias the firefox command.

# If we're forwarding X over SSH, make firefox execute on this machine
if [ -n "$SSH_CONNECTION" -a -n "$DISPLAY" ]; then
    alias firefox='firefox -no-remote'
fi

At least that is what I did.

Archiving your Mail

2009-07-06T00:00:00-07:00

For those that don't know, I use mutt for my email needs. This provides several niceties such as stripping out all the various formatting people like to include in their emails (fonts, graphics, etc), a keyboard driven interface, and, well, it just sucks less that most mail clients.

With mutt I choose to download all my email via POP3 to a local machine where I can read it when I get around to it (rigorous, isn't it). After I read a message and deem it complete I move it to a folder named after the sender (or possibly a group) where I can grep the files and read them at a later date.

However, after a while these files pile up and I need to periodically compress and archive them. This, of course, gets annoying and frequently forgotten. To solve this I needed a script that could parse messages in a number of mail formats, find a date, and determine if it is beyond some threshold at which point it should be archived. These requirements brought me to archivemail. Archivemail supports several input formats (IMAP, mh, mbox, Maildir), archives the messages, and outputs a single mbox formatted file (that can be compresses). While I'm not a huge fan of the mbox format I can easily deal with it for archived mail.

Archivemail has several perks that fit my requirements quite well. First, it was easy to get (packages availables on OS X, Fedora, Debian, and Ubuntu), this probably stems from the fact that it is written in python and can easily run on almost any system. Next, it provides several useful command line options, I personally have a cronjob that archives four message folders every 30 days (logwatches and mail lists) and archives other messages after 180 days. This is simply done with the --days command line switch. I also specify a directory to dump all the archived messages into so they don't clutter up my mail directory. Depending on how you handle you mail there are also options to not archive unread messages or only archive messages older than some fixed date.

For those interested, here is my script that I run as a weekly cronjob to archive and compress my mail messages:

ARCMAIL="/usr/bin/archivemail --quiet --output-dir=$HOME/mail/archive/"

$ARCMAIL --days  30 $HOME/mail/logwatch \
                    $HOME/mail/netflix  \
                    $HOME/mail/amazon   \
                    $HOME/mail/dreamhost

$ARCMAIL --days 180 $HOME/mail/*</pre>

Fairly straightforward, eh?

To search through an archive you can just change into the archive/ directory and execute a gunzip -c <filename> | grep <word>. Alternatively, you can use mutt's built in search and run gunzip <filename>.gz ; mutt -f <filename>.

A Quick Introduction to Makefiles

2009-02-04T00:00:00-08:00

Today at the Marquette Student ACM meeting, I gave a short presentation (PDF) about development on Linux. Specifically using Makefiles. As promised I have uploaded it to this site and I will give a little more information in this post.

Variables

The two main types of variables in a Makefile are "recursively expanded" (=; equal to) and "simply expanded" (:=; set equal to---thanks Algol). Recursively expanded is by far the most common and, realistically, the most confusing. This form of a variable will not perform the substitution until the last possible moment (lazy evaluation), so if you have the line SOURCES = ${CONFIG} demo.c, make will remember that you use ${CONFIG} until it must be known. So if the value of CONFIG changes, the newest version of CONFIG will be used when it is evaluated. Simple expansion occurs when the variable is declared (eager evaluation), if another variable (CFLAGS) is referenced in a simply expanded declaration it will be replaced with the current value of CFLAGS.

Targets

A target occurs on the left-hand side of a rule, typically this is the name of the file you want to build. There are special cases of this, most commonly, "clean". Usually when someone wants to "make clean" they want all the object files generated from a previous make to be removed. However, if you were to create a file named "clean" the rule would never execute because the file clean is up-to-date. This can be remedied with by a PHONY target.

.PHONY: clean
clean:
    rm -f *.o

This creates a phony target that depends on clean, which tells make to ignore any files named clean.

More on Variables

The important "automatic" variables are talked about in the presentation ($@, $<, $^, $+, and $?). Also useful is the % expansion variable. For example, if there was the rule %.o: %.c in a Makefile, this will tell make that to make any file ending in .o will depend on the same file ending in .c (i.e. a rule foo.o: foo.c automatically exists). This will then execute the same commands (say gcc -m32 -Os -o foo.o foo.c) for all files ending in .o. This is a perfect example of why using automatic variables is a great idea.

Well, I think that is everything I have to say about Makefiles in a short amount of time. If you have any questions please feel free to post in the comments. I'll try my best to answer them promptly!

(While most of the content here is off the top of my head, I did reference the GNU make page. They have everything you wanted to know about make.)

./configure --enable-study-mode

2008-02-10T00:00:00-08:00

Well, I'm currently running under the assumption that I have a math test tomorrow. Therefore I should be studying, correct? I assure you I'll get to the point of studying soon. First, however, I want to explain to you how I went from reading notes about birth-and-death processes to updating the ports collection on my FreeBSD box (it won't take long).

So, I begin reading my notes from the beginning of the semester. Nothing looks too bad and I get to the third page in short time, then I start reading about birth-and-death processes. Nothing special, but it does mean a significant change in topics so I decide to take a brief mental break to check my email. No new messages. Back to studying. Wait, I want to start some music---great idea! Let me just load up iTunes. Hmmm, I've already listened to most of this. I know I have some more music on my file server but how to get it to my laptop...?

Well, the easy way would be to copy over the files using something like scp but I don't want to use more disk space on my (already too full) laptop. Ok, I do have NFS set up so I can just mount the music directory and play it over the network. Nah, UDP traffic is for wimps besides it would leave ugly links in iTunes when I leave my network tomorrow. Thinking a few more seconds I realize I want to mount the music directory as if it were someone sharing their iTunes with me. How hard could that be for FreeBSD?

A quick Google brings up the Firefly Media Server that claims to have exactly what I'm looking for. I just need to cd /usr/ports/audio/mt-daapd && make install. D'oh! There is a vulnerability, I need to update the ports tree. So here I am, instead of studying I'm sitting here updating the FreeBSD ports tree. Then I get to build mt-daapd, configure it, and hope that iTunes recognizes it so I can study.

Taking Math Notes on Mac OS X

2008-02-04T00:00:00-08:00

I'll begin with a story. Last week I was taking notes in my mathematics class when the graphite in my mechanical pencil broke. No big deal. Well, not quite, it turns out that the pencil had no more graphite left. So I fall back to my emergency pen with plans to replenish my graphite supply once I get home.

Naturally I forget.

So today when I went to class I decided to bring my computer along (so I could finish modifying my MSCS). Once class begins I reach into my pocket to grab my pencil and realize that I never filled it up. So I think to myself: Should I take notes with my pen or try to go for it with the computer. Being a computer scientist I choose the computer.

The professor begins writing on the chalkboard. . Great. Doesn't that just flow off the fingers. I frantically start trying the various Alt/Alt+Shift key strokes OS X features. No lambda. Alright, maybe someone has already figured this out for me. Nope. The common suggestions are OmniOutliner and NoteTaker. Neither of which flow as naturally as I wanted. Luckily for me I know LaTeX, so I ended up using writing the notes as close to LaTeX style as I could get. I ended up doing pretty well, but I was hoping for something a little easier had been developed. So for Wednesday, I'll probably make myself some shorter commands to allow for a more natural flow when I'm typing my notes.

Anyone know of a decent WYSIWYG program for writing complex mathematical equations on the fly for OS X? I'm alright using LaTeX syntax, I would just like to be able to see the output quickly and on-the-fly.

Automatic backups using `cron` and `tar`

2007-10-06T00:00:00-07:00

This post is an import from a presentation I did in October of 2007. Since I've made this presentation, I've stopped using my own script and suggest you use another tool for backups. I hear rdiff-backup is good. However, I believe this is still a good introduction to bash scripting, cron, and tar.

Original Presentation

Although it may be less useful without the accompanying speaker, the original presentation is available.

Source code for the shell script

I have made the source code (backup.sh (I no longer have this file, sorry -- mjs)) available for download. In the top matter of the file describe how to add the script to your crontab.

Description of the script

For me, the best way to learn something is to take it line by line and that is what I'm going to do below. Naturally I will combine lines which are similar to save space. Since the target audience is someone who has never seen a shell script, some information may seem unimportant to you.

The concatenated source code that appears on this page may not agree with the source available for download. Odds are I decided the change was not worth updating this page, but made available for consumption as the script. If you notice something that is greatly different please contact me.

#!/bin/sh

Selection of a shell interpreter. This must be the first line in the file and be prefix with the 'hash-bang' (or 'sh-bang' for short).

I use /bin/sh since it seems to be the most universal amongst systems. It should be noted that on many systems /bin/sh is that same as /bin/bash, I do not know if this means the script will not work in /bin/sh.

SNAPDIR=/var/snapshots/$USER
RMT_DIR="user@hostname:~/snapshots"
RMT_OPTIONS="-i $HOME/.ssh/id_dsa"

Setting some simple variables. Note that there are no spaces between the variable name, the equal sign, and the value; your script will not work with spaces between these three items.

Here I set the snapshot directory (where snapshots should be stored) to be /var/snapshots/$USER. $USER is a special variable that is the same as the user running the script.

Next are RMT_DIR and RMT_OPTIONS which are quoted. Quotes simply make sure spaces are included in the variable. Again, $HOME is a environmental variable that is always a user's home directory.

RMT_CMD=$(which scp)
DATE=$(date +%Y%m%d)
TAR=$(which tar)
MKDIR=$(which mkdir)
CHMOD=$(which chmod)

This group of variables will run commands in a "sub-shell" before setting the variable name to the value. For example $(which scp) will execute which scp on the system and assign the value returned to RMT_CMD.

LAST_FULL=$(stat -f "%Dc %Sc" -t "%Y%m%d" ${SNAPDIR}/full-*.tar.gz \
            2> /dev/null| sort -n | tail -n1)
LAST_TS=$(echo ${LAST_FULL} | awk '{ print $1}')
LAST_DATE=$(echo ${LAST_FULL} | awk '{ print $2}')

In the final group of variables we use "pipes" which use the output of the first command as the input of the second command. For LAST_FULL we first stat files of the pattern "full-*.tar.gz" in the ${SNAPDIR} directory. (N.B. ${SNAPDIR} dereferences the SNAPDIR variable we set earlier. The curly braces are not strictly necessary, however I use them when referring to local variables.) For stat I am specifying that the output should be of the form " ", then sort the output using the number in the first column, and finally, take only the last file listed.

Once the last full snapshot time is known, we split it into two variables (LAST_TS and LAST_DATE), again using pipes and awk.

if [ ! -d ${SNAPDIR} ]; then
    ${MKDIR} ${SNAPDIR}
    ${CHMOD} go-rwx ${SNAPDIR}
fi

We'll start by making sure the directory snapshots directory exists. If it doesn't, make the directory and remove all permission from anyone not this user.

function incr {
    ${TAR} czf ${SNAPDIR}/incr-${DATE}.tar.gz \
           --exclude-from $HOME/.snap-exclude \
           --listed-incremental=${SNAPDIR}/$USER-${LAST_DATE}.snar \
           $HOME < /dev/null 2< /dev/null

    ${CHMOD} go-rwx ${SNAPDIR}/incr-${DATE}.tar.gz

    if [ "${RMT_CMD}" != "" ]; then
        ${RMT_CMD} ${RMT_OPTIONS} ${SNAPDIR}/incr-${DATE}.tar.gz \
               ${SNAPDIR}/$USER-${LAST_DATE}.snar \
               ${RMT_DIR}
    fi
}

For creating an incremental backup. Using tar, create a gzipped file at ${SNAPDIR}/incr-${DATE}.tar.gz. Since you may not want all of your home directory backed up, you can exclude files listed in the .snap-exclude file. Now the most important part, --listed-incremental, tells tar what the timestamps of the files were last time it executed. If the timestamp on a file is newer than in the snar ("snapshort archive"), it will be added to the tarball. The last argument to tar is simply the directory to backup. > and 2> redirect standard out and standard error to /dev/null, thus suppressing all output.

For the sake of security, we revoke all access from the file except for the current user.

The final step is to check of a RMT_CMD exists, if it does execute it. scp works well for this step, as would rsync.

function full {
    ${TAR} czf ${SNAPDIR}/full-${DATE}.tar.gz \
           --exclude-from $HOME/.snap-exclude \
           --listed-incremental=${SNAPDIR}/$USER-${DATE}.snar \
           $HOME < /dev/null 2< /dev/null

    ${CHMOD} go-rwx ${SNAPDIR}/full-${DATE}.tar.gz
    ${CHMOD} go-rwx ${SNAPDIR}/$USER-${DATE}.snar

    if [ "${RMT_CMD}" != "" ]; then
        ${CMT_CMD} ${RMT_OPTIONS} ${SNAPDIR}/full-${DATE}.tar.gz \
               ${SNAPDIR}/$USER-${DATE}.snar \
               ${RMT_DIR}
    fi
}

Creating a full backup is not much different than an incremental backup. The only difference is the --listed-incremental file (tar will create a new snapshot archive), thus starting with a fresh backup and timestamps. The reason for this is explained in the "Recovery" section.

The rest of the function is mostly the same as an incremental backup.

function normal {
    # Make a full backup if no backup exists
    if [ ! -f ${SNAPDIR}/$USER-${LAST_DATE}.snar ]; then
        full;
    else
        ELAPSED=$(($(date +%s) - ${LAST_TS}))
        SNAP_FRAME=$((7 * 24 * 60 * 60 - 3600))

        # Check if it has been over a week since a full snapshot
        if [ ${ELAPSED} -gt ${SNAP_FRAME} ]; then
            # make a full snapshot
            full;
            # clean up files older than 4 weeks
            clean;
        else
            incr;
        fi

        unset ELAPSED SNAP_FRAME
    fi
}

Here we have the main "brain" of the program. It begins by making sure a backup exists, if one doesn't the script makes a full backup. If at least one full backup exists, then we find out how long it has been since the last full backup and compare that to how frequently full backups should be made. SNAP_FRAME holds the frequency in which backups should be made (every 7 days * 24 hours / day * 60 minutes / hour * 60 seconds / minute (minus 1 hour for time delays)). If too much time has passed, create a full backup and clean out the old files. Otherwise just create an incremental backup.

function clean {
    true
}

The script isn't perfect. I have yet to determine a good way to clean out old files (one that isn't tied to either scp or rsync).

function usage {
    echo "usage: $0 [type]"
    echo "[type] can be one of the following:"
    echo "  normal - follow the daily incremental and weekly backup schedule"
    echo "  incr   - create a incremental backup of $HOME to ${SNAPDIR}"
    echo "  full   - create a full backup of $HOME to ${SNAPDIR}"
    echo "  clean  - cleanup backups older than one month"
    echo "  usage  - display this screen"
    echo "  --help - display this screen"

    exit 1
}

This simple function displays the usage information if requested. $0 is the script name as typed by the user.

case "$1" in
    'normal')
        normal;
        ;;
    'incr')
        incr;
        ;;
    'full')
        full;
        ;;
    'clean')
        clean;
        ;;
    '--help')
        usage;
        ;;
    'usage')
        usage;
        ;;
    *)
        normal;
        ;;
esac

Finally, the driver of the program, a case statement which reads the first argument ($1) and executes the desired function. The default operation is to run in normal mode, but the user is able to force an incremental update, full update, or clean out old files.

Setting up a cronjob

cron is a simple utility that exists on almost all UNIX or UNIX-like systems. A daemon runs every minute to see if any user has a "cronjob" that needs to be executed, if a user does it will run it.

User level cronjobs are maintained by a program called crontab, to view your current crontab type: crontab -l, to edit your crontab use crontab -e.

I personally like to keep all my user level cronjobs in one place, $HOME/.cronjobs/. This folder conatins two files: crontab and backup.sh. crontab is a text file which hold what cronjobs I'd like to have run while backup.sh is the file described above.

My crontab files looks something like this:

# User level crontab
# min hr mday month wday command
00    13  *    *     *    /path/to/home/directory/.cronjobs/backup.sh

Which means I backup my files everyday at precisly 1:00pm by running the file located in /path/to/home/directory/.cronjobs/backup.sh. This can then be loaded into the system cronjobs using the following command.

crontab < crontab

Recovering the data

If you ever need to restore your backed up data all you need to do is find the most recent full backup (we'll say full-20071001.tar.gz) and all the incremental backups since then (in our example incr-2007100[2-5].tar.gz). You'll start by extracting the full backup to the correct folder via tar xzf full-20071001.tar.gz, followed by the incremental backups oldest to newest. Effectively you are restoring your entire home folder from n-days ago and applying the differences from each succeeding day. The commands should go as below (where $ is the shell prompt).

$ tar xzf full-20071001.tar.gz
$ tar xzf incr-20071002.tar.gz
$ tar xzf incr-20071003.tar.gz
$ tar xzf incr-20071004.tar.gz
$ tar xzf incr-20071005.tar.gz

After which you should have you home directory restored exactly as it appeared at 1:00pm on October 10, 2007.

Beyond Syntax | Michael J. Schultz

On Demand ssh-add

Background

What is a geek to do?

Sophos Holidy Puzzle Walkthrough

Step 1

Step 2

Diving into the Linux Networking Stack, Part I

Starting at the Driver

A General View of Network Drivers

A Quick Refresher on Driver Basics

A Concrete Example in igb

Initialization

Opening the Device

Allocating Resources

Configuring Interrupts

Receiving a Packet

Conclusions

Performance Monitoring with OProfile

Event Counters

Event Ratios

Using oprofile

opcontrol

opreport

A Second Example

Conclusions

The nth Backup Solution

Mozilla Fun

Technology and Courage

RSS Sucks.

Remote Instance of Firefox via SSH -X

Archiving your Mail

A Quick Introduction to Makefiles

Variables

Targets

More on Variables

./configure --enable-study-mode

Taking Math Notes on Mac OS X

Automatic backups using `cron` and `tar`

Original Presentation

Source code for the shell script

Description of the script

Setting up a cronjob

Recovering the data

A Concrete Example in `igb`

Using `oprofile`

`opcontrol`

`opreport`