   This file documents GNU `tar', a utility used to store, backup, and
transport files.

   Copyright (C) 1992, 1994, 1995 Free Software Foundation, Inc.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.

The Standard Format
===================

     *(This message will disappear, once this node revised.)*

   A `tar' archive file contains a series of records.  Each record
contains `RECORDSIZE' bytes.  Although this format may be thought of as
being on magnetic tape, other media are often used.

   Each file archived is represented by a header record which describes
the file, followed by zero or more records which give the contents of
the file.  At the end of the archive file there may be a record filled
with binary zeros as an end-of-file marker.  A reasonable system should
write a record of zeros at the end, but must not assume that such a
record exists when reading an archive.

   The records may be "blocked" for physical I/O operations.  Each
block of N records (where N is set by the `--block-size=512-SIZE' (`-b
512-SIZE') option to `tar') is written with a single `write ()'
operation.  On magnetic tapes, the result of such a write is a single
tape record.  When writing an archive, the last block of records should
be written at the full size, with records after the zero record
containing all zeroes.  When reading an archive, a reasonable system
should properly handle an archive whose last block is shorter than the
rest, or which contains garbage records after a zero record.

   The header record is defined in C as follows.  In the GNU `tar'
distribution, this is part of file `src/tar.h':

     /* Standard Archive Format - Standard TAR - USTAR.  */

     /* Header block on tape.

        We use traditional DP naming conventions here.  A "block" is a big chunk
        of stuff that we do I/O on.  A "record" is a piece of info that we care
        about.  Typically many "record"s fit into a "block".  */

     #define RECORDSIZE      512
     #define NAMSIZ          100
     #define TUNMLEN         32
     #define TGNMLEN         32
     #define SPARSE_EXT_HDR  21
     #define SPARSE_IN_HDR   4

     struct sparse
       {
         char offset[12];
         char numbytes[12];
       };

     union record
       {
         char charptr[RECORDSIZE];

         struct header
           {
             char arch_name[NAMSIZ];
             char mode[8];
             char uid[8];
             char gid[8];
             char size[12];
             char mtime[12];
             char chksum[8];
             char linkflag;
             char arch_linkname[NAMSIZ];
             char magic[8];
             char uname[TUNMLEN];
             char gname[TGNMLEN];
             char devmajor[8];
             char devminor[8];

             /* The following fields were added for GNU and are not standard.  */

             char atime[12];
             char ctime[12];
             char offset[12];
             char longnames[4];
             /* Some compilers would insert the pad themselves, so pad was
                once autoconfigured.  It is simpler to always insert it!  */
             char pad;
             struct sparse sp[SPARSE_IN_HDR];
             char isextended;
             char realsize[12];      /* true size of the sparse file */
     #if 0
             char ending_blanks[12]; /* number of nulls at the end of the file,
                                        if any */
     #endif
           }
         header;

         struct extended_header
           {
             struct sparse sp[21];
             char isextended;
           }
         ext_hdr;
       };

     /* The checksum field is filled with this while the checksum is computed.  */
     #define CHKBLANKS       "        "      /* 8 blanks, no null */

     /* The magic field is filled with this value if uname and gname are valid,
        marking the archive as being in standard POSIX format (though GNU tar
        itself is not POSIX conforming).  */
     #define TMAGIC "ustar  "        /* 7 chars and a null */

     /* The magic field is filled with this if this is a GNU format dump entry.
        But I suspect this is not true anymore.  */
     #define GNUMAGIC "GNUtar "      /* 7 chars and a null */

     /* The linkflag defines the type of file.  */
     #define LF_OLDNORMAL    '\0'    /* normal disk file, Unix compat */
     #define LF_NORMAL       '0'     /* normal disk file */
     #define LF_LINK         '1'     /* link to previously dumped file */
     #define LF_SYMLINK      '2'     /* symbolic link */
     #define LF_CHR          '3'     /* character special file */
     #define LF_BLK          '4'     /* block special file */
     #define LF_DIR          '5'     /* directory */
     #define LF_FIFO         '6'     /* FIFO special file */
     #define LF_CONTIG       '7'     /* contiguous file */
     /* Further link types may be defined later.  */

     /* Note that the standards committee allows only capital A through
        capital Z for user-defined expansion.  This means that defining
        something as, say '8' is a *bad* idea.  */

     /* This is a dir entry that contains the names of files that were in the
        dir at the time the dump was made.  */
     #define LF_DUMPDIR      'D'

     /* Identifies the NEXT file on the tape as having a long linkname.  */
     #define LF_LONGLINK     'K'

     /* Identifies the NEXT file on the tape as having a long name.  */
     #define LF_LONGNAME     'L'

     /* This is the continuation of a file that began on another volume.  */
     #define LF_MULTIVOL     'M'

     /* For storing filenames that didn't fit in 100 characters.  */
     #define LF_NAMES        'N'

     /* This is for sparse files.  */
     #define LF_SPARSE       'S'

     /* This file is a tape/volume header.  Ignore it on extraction.  */
     #define LF_VOLHDR       'V'

     #if 0
     /* The following two blocks of #define's are unused in GNU tar.  */

     /* Bits used in the mode field - values in octal */
     #define  TSUID    04000         /* set UID on execution */
     #define  TSGID    02000         /* set GID on execution */
     #define  TSVTX    01000         /* save text (sticky bit) */

     /* File permissions */
     #define  TUREAD   00400         /* read by owner */
     #define  TUWRITE  00200         /* write by owner */
     #define  TUEXEC   00100         /* execute/search by owner */
     #define  TGREAD   00040         /* read by group */
     #define  TGWRITE  00020         /* write by group */
     #define  TGEXEC   00010         /* execute/search by group */
     #define  TOREAD   00004         /* read by other */
     #define  TOWRITE  00002         /* write by other */
     #define  TOEXEC   00001         /* execute/search by other */

     #endif

     /* End of Standard Archive Format description.  */

   All characters in header records are represented by using 8-bit
characters in the local variant of ASCII.  Each field within the
structure is contiguous; that is, there is no padding used within the
structure.  Each character on the archive medium is stored contiguously.

   Bytes representing the contents of files (after the header record of
each file) are not translated in any way and are not constrained to
represent characters in any character set.  The `tar' format does not
distinguish text files from binary files, and no translation of file
contents is performed.

   The `name', `linkname', `magic', `uname', and `gname' are
null-terminated character strings.  All other fileds are zero-filled
octal numbers in ASCII.  Each numeric field of width W contains W minus
2 digits, a space, and a null, except `size', and `mtime', which do not
contain the trailing null.

   The `name' field is the file name of the file, with directory names
(if any) preceding the file name, separated by slashes.
FIXME: how big a name before field overflows?

   The `mode' field provides nine bits specifying file permissions and
three bits to specify the Set UID, Set GID, and Save Text ("sticky")
modes.  Values for these bits are defined above.  When special
permissions are required to create a file with a given mode, and the
user restoring files from the archive does not hold such permissions,
the mode bit(s) specifying those special permissions are ignored.
Modes which are not supported by the operating system restoring files
from the archive will be ignored.  Unsupported modes should be faked up
when creating or updating an archive; e.g. the group permission could
be copied from the *other* permission.

   The `uid' and `gid' fields are the numeric user and group ID of the
file owners, respectively.  If the operating system does not support
numeric user or group IDs, these fields should be ignored.

   The `size' field is the size of the file in bytes; linked files are
archived with this field specified as zero.
FIXME: xref Modifiers
   , in particular the `--incremental' (`-G') option.

   The `mtime' field is the modification time of the file at the time
it was archived.  It is the ASCII representation of the octal value of
the last time the file was modified, represented as an integer number of
seconds since January 1, 1970, 00:00 Coordinated Universal Time.

   The `chksum' field is the ASCII representation of the octal value of
the simple sum of all bytes in the header record.  Each 8-bit byte in
the header is added to an unsigned integer, initialized to zero, the
precision of which shall be no less than seventeen bits.  When
calculating the checksum, the `chksum' field is treated as if it were
all blanks.

   The `typeflag' field specifies the type of file archived.  If a
particular implementation does not recognize or permit the specified
type, the file will be extracted as if it were a regular file.  As this
action occurs, `tar' issues a warning to the standard error.

   The `atime' and `ctime' fields are used in making incremental
backups; they store, respectively, the particular file's access time
and last inode-change time.

   The `offset' is used by the `--multi-volume' (`-M') option, when
making a multi-volume archive.  The offset is number of bytes into the
file that we need to restart at to continue the file on the next tape,
i.e., where we store the location that a continued file is continued at.

   The following fields were added to deal with sparse files.  A file
is "sparse" if it takes in unallocated blocks which end up being
represented as zeros, i.e., no useful data.  A test to see if a file is
sparse is to look at the number blocks allocated for it versus the
number of characters in the file; if there are fewer blocks allocated
for the file than would normally be allocated for a file of that size,
then the file is sparse.  This is the method `tar' uses to detect a
sparse file, and once such a file is detected, it is treated
differently from non-sparse files.

   Sparse files are often `dbm' files, or other database-type files
which have data at some points and emptiness in the greater part of the
file.  Such files can appear to be very large when an `ls -l' is done
on them, when in truth, there may be a very small amount of important
data contained in the file.  It is thus undesirable to have `tar' think
that it must back up this entire file, as great quantities of room are
wasted on empty blocks, which can lead to running out of room on a tape
far earlier than is necessary.  Thus, sparse files are dealt with so
that these empty blocks are not written to the tape.  Instead, what is
written to the tape is a description, of sorts, of the sparse file:
where the holes are, how big the holes are, and how much data is found
at the end of the hole.  This way, the file takes up potentially far
less room on the tape, and when the file is extracted later on, it will
look exactly the way it looked beforehand.  The following is a
description of the fields used to handle a sparse file:

   The `sp' is an array of `struct sparse'.  Each `struct sparse'
contains two 12-character strings which represent an offset into the
file and a number of bytes to be written at that offset.  The offset is
absolute, and not relative to the offset in preceding array element.

   The header can hold four of these `struct sparse' at the moment; if
more are needed, they are not stored in the header.

   The `isextended' flag is set when an `extended_header' is needed to
deal with a file.  Note that this means that this flag can only be set
when dealing with a sparse file, and it is only set in the event that
the description of the file will not fit in the alloted room for sparse
structures in the header.  In other words, an extended_header is needed.

   The `extended_header' structure is used for sparse files which need
more sparse structures than can fit in the header.  The header can fit
4 such structures; if more are needed, the flag `isextended' gets set
and the next record is an `extended_header'.

   Each `extended_header' structure contains an array of 21 sparse
structures, along with a similar `isextended' flag that the header had.
There can be an indeterminate number of such `extended_header's to
describe a sparse file.

`LF_NORMAL'
`LF_OLDNORMAL'
     These flags represent a regular file.  In order to be compatible
     with older versions of `tar', a `typeflag' value of `LF_OLDNORMAL'
     should be silently recognized as a regular file.  New archives
     should be created using `LF_NORMAL'.  Also, for backward
     compatibility, `tar' treats a regular file whose name ends with a
     slash as a directory.

`LF_LINK'
     This flag represents a file linked to another file, of any type,
     previously archived.  Such files are identified in Unix by each
     file having the same device and inode number.  The linked-to name
     is specified in the `linkname' field with a trailing null.

`LF_SYMLINK'
     This represents a symbolic link to another file.  The linked-to
     name is specified in the `linkname' field with a trailing null.

`LF_CHR'
`LF_BLK'
     These represent character special files and block special files
     respectively.  In this case the `devmajor' and `devminor' fields
     will contain the major and minor device numbers respectively.
     Operating systems may map the device specifications to their own
     local specification, or may ignore the entry.

`LF_DIR'
     This flag specifies a directory or sub-directory.  The directory
     name in the `name' field should end with a slash.  On systems where
     disk allocation is performed on a directory basis, the `size' field
     will contain the maximum number of bytes (which may be rounded to
     the nearest disk block allocation unit) which the directory may
     hold.  A `size' field of zero indicates no such limiting.  Systems
     which do not support limiting in this manner should ignore the
     `size' field.

`LF_FIFO'
     This specifies a FIFO special file.  Note that the archiving of a
     FIFO file archives the existence of this file and not its contents.

`LF_CONTIG'
     This specifies a contiguous file, which is the same as a normal
     file except that, in operating systems which support it, all its
     space is allocated contiguously on the disk.  Operating systems
     which do not allow contiguous allocation should silently treat this
     type as a normal file.

`A' ... `Z'
     These are reserved for custom implementations.  Some of these are
     used in the GNU modified format, as described below.

   Other values are reserved for specification in future revisions of
the P1003 standard, and should not be used by any `tar' program.

   The `magic' field indicates that this archive was output in the
P1003 archive format.  If this field contains `TMAGIC', the `uname' and
`gname' fields will contain the ASCII representation of the owner and
group of the file respectively.  If found, the user and group IDs are
used rather than the values in the `uid' and `gid' fields.

GNU Extensions to the Archive Format
====================================

     *(This message will disappear, once this node revised.)*

   The GNU format uses additional file types to describe new types of
files in an archive.  These are listed below.

`LF_DUMPDIR'
`'D''
     This represents a directory and a list of files created by the
     `--incremental' (`-G') option.  The `size' field gives the total
     size of the associated list of files.  Each file name is preceded
     by either a `Y' (the file should be in this archive) or an `N'.
     (The file is a directory, or is not stored in the archive.)  Each
     file name is terminated by a null.  There is an additional null
     after the last file name.

`LF_MULTIVOL'
`'M''
     This represents a file continued from another volume of a
     multi-volume archive created with the `--multi-volume' (`-M')
     option.  The original type of the file is not given here.  The
     `size' field gives the maximum size of this piece of the file
     (assuming the volume does not end before the file is written out).
     The `offset' field gives the offset from the beginning of the
     file where this part of the file begins.  Thus `size' plus
     `offset' should equal the original size of the file.

`LF_SPARSE'
`'S''
     This flag indicates that we are dealing with a sparse file.  Note
     that archiving a sparse file requires special operations to find
     holes in the file, which mark the positions of these holes, along
     with the number of bytes of data to be found after the hole.

`LF_VOLHDR'
`'V''
     This file type is used to mark the volume header that was given
     with the `--label=ARCHIVE-LABEL' (`-V ARCHIVE-LABEL') option when
     the archive was created.  The `name' field contains the `name'
     given after the `--label=ARCHIVE-LABEL' (`-V ARCHIVE-LABEL')
     option.  The `size' field is zero.  Only the first file in each
     volume of an archive should have this type.

   You may have trouble reading a GNU format archive on a non-GNU
system if the options `--incremental' (`-G'), `--multi-volume' (`-M'),
`--sparse' (`-S'), or `--label=ARCHIVE-LABEL' (`-V ARCHIVE-LABEL') were
used when writing the archive.  In general, if `tar' does not use the
GNU-added fields of the header, other versions of `tar' should be able
to read the archive.  Otherwise, the `tar' program will give an error,
the most likely one being a checksum error.

Comparison of `tar' and `cpio'
==============================

     *(This message will disappear, once this node revised.)*

   Here is a summary of differences between `tar' and `cpio'.  The
accuracy of the following information has not been verified.  The
following people contributed to this section, mainly through a survey
conducted in 1991.  The remainder of this section does not otherwise
try to relate topics to people.

     Bent Bertelsen          dmdata@login.dkuug.dk
     David Hoopes            talgras!david
     Guy Harris              guy@auspex.com
     Kai Petzke              wpp@marie.physik.tu-berlin.de
     Kristen Nielsen         dmdata@login.dkuug.dk
     Leslie Mikesell         les@chinet.chi.il.us

FIXME: Reorganize the following material

   `tar' handles symbolic links in the form in which it comes in BSD;
`cpio' doesn't handle symbolic links in the form in which it comes in
System V prior to SVR4, and some vendors may have added symlinks to
their system without enhancing `cpio' to know about them.  Others may
have enhanced it in a way other than the way I did it at Sun, and which
was adopted by AT&T (and which is, I think, also present in the `cpio'
that Berkeley picked up from AT&T and put into a later BSD release--I
think I gave them my changes).

   (SVR4 does some funny stuff with `tar'; basically, its `cpio' can
handle `tar' format input, and write it on output, and it probably
handles symbolic links.  They may not have bothered doing anything to
enhance `tar' as a result.)

   `cpio' handles special files; traditional `tar' doesn't.

   `tar' comes with V7, System III, System V, and BSD source; `cpio'
comes only with System III, System V, and later BSD (4.3-tahoe and
later).

   `tar''s way of handling multiple hard links to a file can handle
file systems that support 32-bit inumbers (e.g., the BSD file system);
`cpio's way requires you to play some games (in its "binary" format,
i-numbers are only 16 bits, and in its "portable ASCII" format, they're
18 bits--it would have to play games with the "file system ID" field of
the header to make sure that the file system ID/i-number pairs of
different files were always different), and I don't know which `cpio's,
if any, play those games.  Those that don't might get confused and
think two files are the same file when they're not, and make hard links
between them.

   `tar's way of handling multiple hard links to a file places only one
copy of the link on the tape, but the name attached to that copy is the
*only* one you can use to retrieve the file; `cpio's way puts one copy
for every link, but you can retrieve it using any of the names.

     >What type of check sum (if any) is used, and how is this
     calculated.

   See the attached manual pages for `tar' and `cpio' format.  `tar'
uses a checksum which is the sum of all the bytes in the `tar' header
for a file; `cpio' uses no checksum.

     >If anyone knows why `cpio' was made when `tar' was prasent >at
     the unix scene,

   It wasn't.  `cpio' first showed up in PWB/UNIX 1.0; no
generally-available version of UNIX had `tar' at the time.  I don't
know whether any version that was generally available *within AT&T* had
`tar', or, if so, whether the people within AT&T who did `cpio' knew
about it.

   On restore, if there is a corruption on a tape `tar' will stop at
that point, while `cpio' will skip over it and try to restore the rest
of the files.

   The main difference is just in the command syntax and header format.

   `tar' is a little more tape-oriented in that everything is blocked
to start on a block boundary.

     >Is there any differences between the ability to recover crashed
     >archives between the two of them. (Is there any chance of
     recovering >crashed archives at all.)

   Theoretically it should be easier under `tar' since the blocking
lets you find a header with some variation of `dd skip=NN'.  However,
modern `cpio''s and variations have an option to just search for the
next file header after an error with a reasonable chance of re-syncing.
However, lots of tape driver software won't allow you to continue past
a media error which should be the only reason for getting out of sync
unless a file changed sizes while you were writing the archive.

     >If anyone knows why `cpio' was made when `tar' was prasent >at
     the unix scene, please tell me about this too.

   Probably because it is more media efficient (by not blocking
everything and using only the space needed for the headers where `tar'
always uses 512 bytes per file header) and it knows how to archive
special files.

   You might want to look at the freely available alternatives.  The
major ones are `afio', GNU `tar', and `pax', each of which have their
own extensions with some backwards compatibility.

   Sparse files were `tar'red as sparse files (which you can easily
test, because the resulting archive gets smaller, and GNU `cpio' can no
longer read it).