Command Line Too Long

I spent the weekend writing scripts to handle various situations where a very long command line was necessary. Usually it was handling one command that had to be repeated for each file in a directory.

For example, pkg_add on FreeBSD doesn't handle more than 200 packages at a time on the command line. So I spent the first part of the evening writing little scripts like:

# sh
 for pkg in *
 do
 pkg_add $pkg
 done

Those scripts work, but later I found a utility designed just for this particular difficulty: xargs. 

According to the man page:

The xargs utility reads space, tab, newline and end-of-file delimited arguments
from the standard input and executes the specified utility with them as
arguments.

Basically it means you pipe a list of "arguments" to xargs. Xargs sucks up the list and then feeds them back one (or more) at a time to the specified program until they are all gone. For my application, this worked like a charm. My already small script was reduced to:

ls | xargs  -n1 pkg_add

The ls command produces a list of files in the directory, which I can modify a bit more readily than in the shell script, and feeds it via a pipe to the xargs command. The -n1 option tells it to feed pkg_add only one filename at a time.

That handled my bulk pkg_add problem and I moved on to other issues that needed xargs. I spent some time debugging an xargs command written by someone else to do font conversions that was having some unexpected results.

The original command took font files on Mac OS X and converted them to TTF files using the fondu command. Since you never knew how many fonts a person would have, the xargs command made feeding fondu one font at a time very nice. The original command looked like this:

ls | xargs -n1 /usr/local/bin/fondu -force

From the first glance this looks quite harmless and almost exactly like the other one. In fact, for the most part this did exactly what was expected. In a folder with only a few fonts, the command:

/usr/local/bin/fondu -force *

would work just fine. So, I tested both commands and the non-xargs command actually produced more converted fonts. This anomoly had to have something to do with the way xargs was feeding the font list to fondu.

To make things a bit more clear as I ran the script, I added the -t option to xargs.

-t      Echo the command to be executed to standard error immediately
        before it is executed.

So the command looked something like this:

ls | xargs -n1 /usr/local/bin/fondu -force

I quickly became aware of the nature of the problem. According to the man page description, xargs uses whitespace to delimit or separate arguments. So, some of the font files had spaces in the name and xargs was passing them individually to fondu for processing. A bit more reading in the man page and I was able to find the answer.

-0      Use NUL (``\0'') instead of whitespace as the argument 
             separator.  This can be used in conjuction with 
             the -print0 option of find(1).

The -0 (numeric zero) option told xargs to pass the entire line to fondu for processing as one argument. This kept files with spaces in their names from being broken up into separate commands and fixed my problem. The new command looked like this:

ls | xargs -0 -n1 /usr/local/bin/fondu -force

The fonts were properly converted and everyone was happy.

The -0 option man page entry makes reference to the -print0 option in the find utility. When looking for files, find often comes in handy. Here is a quick hack to give details on all the True Type fonts on your system:

find / -name "*.ttf" -print0 | xargs -0 ls -l

This system works well as long as the arguments fed to xargs only come at the end of the command line, because xargs only appends. It can't insert into the middle. However on FreeBSD, they have added support for just that. The -J option sets the placeholder character for xargs to insert the given arguments at.

For example:

find / -name "*.ttf" -print0 | xargs -J % -0 cp -rp % /usr/home/newfonts

The above command would find all the TTF fonts on your system and copy them to /usr/home/newfonts. If you used the mv command it would be a nice way to consolidate one type of files into a common folder.

The man page for Mac OS X suggests that it has also picked up this enhanced version of xargs, but I haven't tested it yet.

This should get you through the basics of xargs and well on your way to some of the more advanced xargs tricks.