Linux - Using tar for archive files

[josephj]

Some people at the Linux SIG expressed an interest in using tar with Linux.

The rest of this article assumes basic familiarity with using the command line in Linux.  A simple way to get to the command line from inside your gui is to open a console program like konsole or xterm from your Start Applications Menu.

tar is a very handy program.  It was originally developed for creating archives of files on magnetic tape for backups, etc.  Nowadays, it is used primarily as a simple method to package a group of files into one file for ease of distribution.  Many Linux "source" packages come in this format.  It is also handy for making simple compressed backup archives.

When it's created "right" (IMHO), a tar archive consists of a single directory with files (and possibly subdirectories) in it so that if you unpack it using default options, what you get is a new subdirectory with all the separate files in it.  This way you know exactly where all of the parts are and they're not mixed in with anything else.

In addition to creating archives and extracting files from them, tar can also use one of several compression programs to shrink the size of the archive.  The two most common compression types are zip and bzip indicated by using the z and j flags respectively.

Usually, a tar archive is named some-name.tar.
If it was zipped it is named some-name.tar.gz.
If it was bzipped, it is named some-name.tar.bz.

Of course, this is the wild woolly web and people do what they will, so the above is not a hard and fast rule.  If you're not sure what you've got, type

file some-file

and it will tell you what kind of file some-file (probably) is.

So, if something you want comes in a tar archive, you (put it or) download it into a working directory as a regular user (I use /home/bigbird/installs for good stuff and /home/bigbird/sandbox for temporary things or ones I'm not sure about).  Then, you extract all or some of the files.  Once the files are extracted, you need to put them where they'll do some good.

If they comprise a program, you usually run configure and make scripts that (hopefully) come with the package and then, as superuser (su or sudo), run a make install to put the files in the right protected places in your operating system.  To cover this adequately would require a book chapter, but the above steps often work as is.

If the files are not a program or are too simple to require all the work of creating configure, make, and install scripts, you just move them to where you want them on your computer and then make sure they have the right permissions set using chmod if necessary.

If any of the files are executable (e.g. scripts or binary programs), you need to put them where the rest of your system can find them.  Some files will need to be put in special places e.g. .jar files need to go where your java systems know to look for them.

Simple executable files like shell scripts or binary programs usually need to be in a directory that is searched by the operating system when you tell it to do something.   In Linux, this information is stored in the shell variable PATH.  At the command line, type
echo $PATH
and you'll see where Linux is looking.

bigbird ~: $echo $PATH
/usr/local/qt-3.3.2/bin:/usr//bin:/bin:/usr/bin::/usr/local/bin:/usr/X11R6/bin:/usr/games:/home/bigbird/bin
bigbird ~: $

I keep all my personal programs in /home/bigbird/bin and I have added that to the end of my PATH so that my programs will be found by the system, but will not override any program of the same name already present in the system.

Back to tar:

tar, like many programs in Linux, has a ton of options, but most of them are for special cases.

To extract a tar file into the current directory:
tar -xvf some-name.tar

To extract a tar.gz file into the current directory:
tar -xvzf some-name.tar.gz

To extract a tar.bz file into the current directory:
tar -xvjf some-name.tar.bz

See
http://www.fluidthoughts.com/howto/tar-gzip/

for a few more details or type

man tar

for the whole enchilada.

TIP:

Another use for tar that's a little less obvious is to use it as a copy program for moving groups of files around on your disk(s) or network.  Using tar alone or with pipes in a shell script can be a handy way to package up a pile of files on the fly and send them on their way as a single job step - possibly using file compression to cut down the bandwidth/transmission time requirements.

If you're doing a lot of this, take a look at the rsync command.  It's great for copying just what has changed between two locations.

HTH

Joe

[Relic]

Sounds like a good meeting topic from start to finish.

[fpirrone]

Nice post, Joe, thanks for sharing, and yes Relic, we will visit the CLI probably at the next meeting.  If Joe is there he can address this crucial step in performing typical compiles from the CLI, which we will also explore next time.

We'll take a look at file permissions at the same time, and Joe's practice of doing installs to a /home/bigbird/bin gets around the problem of/need to "sudo make install" to drop self-compiled applications into /usr/local/bin.  There are also implications for a multi-user system in Joe's and the alternate approaches.

One other quick comment, that file utility, Midnight Commander, can also un-gzip or un-bzip and un-tar directly, and can even "open up" .deb and other package files...among about a million other things.

I'll probably wind up folding in a review of MC's core functionality as the context for the above, taking a look at the typical GNU/Linux filesystem in the process.

Should be a rip-roaring session, and nobody should get caught without their propeller beanie...

Frank

[josephj]

Just to clarify:

I install most things using make install or, failing that, putting them into /usr/bin, /usr/local/bin, or occasionally, /opt.  That way all users have access to them and I don't run into the problems that Frank alludes to.

I use ~/bin for experimental stuff and for the occasional package where I can't figure out the permissions, etc. and just want to get it to work.

When I do it this way, the package is only available for my user, but on my system, which is essentially a single user system, that's not a problem.

This is Linux.  There are almost always lots of ways to accomplish a task and each one comes with its own set of pros and cons.

[fpirrone]

Yeah, Joe, that's exactly what I meant.  For the sake of those newbies reading this little discussion thread, let me lay out the basic self-compile issue, as it provokes both critique and anxiety, generally needlessly:

First, most source code versions of applications come, as Joe explained in detail, as tarred and zipped files.  Tar stands for "tape archive" and its purpose was to "bundle" up a group of files, generally for archiving, and gzip "squeezes the air" out of that bundle while bzip squeezes a little harder.  In a very large source file, the bzipped version may be megabytes smaller than the gzipped one.

So, step one is to unzip that source to perform the compilation, and as Joe said:

tar -xvzf whatever

will do that for gzipped files (named, whatever).

step 2 is to simply prepare the source code for compiling by moving into the folder created by the above step, and configuring it for your system, gathering up all the headers and libraries that it calls for:

cd source_code_folder_whatever_i t_was_named
./configure

If you want to create the executable program in a particular location where executables are found, you simply add the --prefix argument.  Now, with putting your self-compiled programs in your own private binary directory, that supports not only explicitly backing up those "special" things, and if desired employing a separate partition for all user directories under /home, but also keeping your special things completely under the control of your own account. 

It also supports, on a multi-user system, having different versions of the same program for each person needing that.  DONT try that with Windows!  Since each user's path statement contains their own binary directory, when they create a program launcher or execute that application from a CLI, they get their own version.

So, step two becomes, for the above approach:

cd whatever
./configure --prefix=/home/bigbird/bin

or, to put a single version of that program into the local binary directory where all users could access it:

cd whatever
./configure --prefix=/usr/local/

which is the most common default location, and the one that would therefore be used if you left off that argument.

You, of course could also "force" your self-compiled binary to be placed into the /usr/bin location where most package binaries are found, and yes, you're right, that would just require:

cd whatever
./configure --prefix=/usr

A consequence of placing self-compiled binaries there is that they'll mix with package binaries.  That's not a problem, but ever so slightly messy.  A consequence of placing self-compiled binaries in either /usr/local/bin or /usr/bin is that those subdirectories are already in the path statement, so you don't need to edit that.  Obviously, adding /home/bigbird/bin into your path will take all of 2 seconds - 1 if it was just bird(!), so it's no big deal in any event.

The third step is to execute:

make

which causes the source code to be compiled according to whatever is in its makefile and what you may have added during the configure

And, finally, the fourth and final step is to:

sudo make install

if you configured the application to be placed in one of the two system locations - /usr/bin or /usr/local/bin, since writing to those locations requires administrative privileges.  If you set it to write to your private binary directory, the last step is simply:

make install.

So, that wasn't too bad, was it?  Showing it as it would actually look at the CLI in a little terminal window open on your desktop:

tar -xvzf whatever.tar.gz
cd whatever
./configure --prefix=/usr/local
make
sudo make install

If your system has all the support files required to do compiles, that's probably all it will take for most common compiles.  What you lost was a few minutes of time compared with installing from a package using a graphical or even a CLI package manager, but what you gained was possibly a program that wasn't available as a package, possibly a program that was up-to-the-minute current, and definitely a program that was custom-built right for and right on your very computer.

Questions?

Frank

[josephj]

As long as. this has turned into a thread on installing ...

Many tarball packages include source files in c, c++, etc. and have commands that will compile them in the make file that comes with them.  make will try to take care of this all automagically, but your system has to be set up to support it.

ubuntu-based distros (at least kubuntu) are so end-user oriented that they don't come set up to do compiles like this.

If this is the case, you'll get error messages like cpp not found or gcc not found when you run configure or make.

In ubuntu, this is easy to fix.  Just go into your package manager (apt, synaptic, kpackagekit, or adept) and install the meta-package called build-essential.

This is a package that doesn't contain any code, but is configured to depend on a list of packages that have all the things a typical package needs to compile and install correctly.  Because of these dependencies, these packages and any they depend on all get installed just by installing this one package.  It's analogous to a macro in other applications.

If you're using a distro that's not in the ubuntu family, then you'll have to find an equivalent procedure or just keep running the configure and make and then installing whatever packages it complains about not finding until you get them all.