1 SQUASHFS 4.3 - A squashed read-only filesystem for Linux
2
3 Copyright 2002-2014 Phillip Lougher <phillip@lougher.demon.co.uk>
4
5 Released under the GPL licence (version 2 or later).
6
7Welcome to Squashfs version 4.3. Please read the README-4.3 and CHANGES files
8for details of changes.
9
10Squashfs is a highly compressed read-only filesystem for Linux.
11It uses either gzip/xz/lzo/lz4 compression to compress both files, inodes
12and directories. Inodes in the system are very small and all blocks are
13packed to minimise data overhead. Block sizes greater than 4K are supported
14up to a maximum of 1Mbytes (default block size 128K).
15
16Squashfs is intended for general read-only filesystem use, for archival
17use (i.e. in cases where a .tar.gz file may be used), and in constrained
18block device/memory systems (e.g. embedded systems) where low overhead is
19needed.
20
211. SQUASHFS OVERVIEW
22--------------------
23
241. Data, inodes and directories are compressed.
25
262. Squashfs stores full uid/gids (32 bits), and file creation time.
27
283. In theory files up to 2^64 bytes are supported. In theory filesystems can
29 be up to 2^64 bytes.
30
314. Inode and directory data are highly compacted, and packed on byte
32 boundaries. Each compressed inode is on average 8 bytes in length
33 (the exact length varies on file type, i.e. regular file, directory,
34 symbolic link, and block/char device inodes have different sizes).
35
365. Squashfs can use block sizes up to 1Mbyte (the default size is 128K).
37 Using 128K blocks achieves greater compression ratios than the normal
38 4K block size.
39
406. File duplicates are detected and removed.
41
427. Filesystems can be compressed with gzip, xz (lzma2), lzo or lz4
43 compression algorithms.
44
451.1 Extended attributes (xattrs)
46--------------------------------
47
48Squashfs filesystems now have extended attribute support. The
49extended attribute implementation has the following features:
50
511. Layout can store up to 2^48 bytes of compressed xattr data.
522. Number of xattrs per inode unlimited.
533. Total size of xattr data per inode 2^48 bytes of compressed data.
544. Up to 4 Gbytes of data per xattr value.
555. Inline and out-of-line xattr values supported for higher performance
56 in xattr scanning (listxattr & getxattr), and to allow xattr value
57 de-duplication.
586. Both whole inode xattr duplicate detection and individual xattr value
59 duplicate detection supported. These can obviously nest, file C's
60 xattrs can be a complete duplicate of file B, and file B's xattrs
61 can be a partial duplicate of file A.
627. Xattr name prefix types stored, allowing the redundant "user.", "trusted."
63 etc. characters to be eliminated and more concisely stored.
648. Support for files, directories, symbolic links, device nodes, fifos
65 and sockets.
66
67Extended attribute support is in 2.6.35 and later kernels. Filesystems
68with extended attributes can be mounted on 2.6.29 and later kernels, the
69extended attributes will be ignored with a warning.
70
712. USING SQUASHFS
72-----------------
73
74Squashfs filesystems should be mounted with 'mount' with the filesystem type
75'squashfs'. If the filesystem is on a block device, the filesystem can be
76mounted directly, e.g.
77
78%mount -t squashfs /dev/sda1 /mnt
79
80Will mount the squashfs filesystem on "/dev/sda1" under the directory "/mnt".
81
82If the squashfs filesystem has been written to a file, the loopback device
83can be used to mount it (loopback support must be in the kernel), e.g.
84
85%mount -t squashfs image /mnt -o loop
86
87Will mount the squashfs filesystem in the file "image" under
88the directory "/mnt".
89
903. MKSQUASHFS
91-------------
92
933.1 Mksquashfs options and overview
94-----------------------------------
95
96As squashfs is a read-only filesystem, the mksquashfs program must be used to
97create populated squashfs filesystems.
98
99SYNTAX:./mksquashfs source1 source2 ... dest [options] [-e list of exclude
100dirs/files]
101
102Filesystem build options:
103-comp <comp> select <comp> compression
104 Compressors available:
105 gzip (default)
106 lzo
107 lz4
108 xz
109-b <block_size> set data block to <block_size>. Default 128 Kbytes
110 Optionally a suffix of K or M can be given to specify
111 Kbytes or Mbytes respectively
112-no-exports don't make the filesystem exportable via NFS
113-no-sparse don't detect sparse files
114-no-xattrs don't store extended attributes
115-xattrs store extended attributes (default)
116-noI do not compress inode table
117-noD do not compress data blocks
118-noF do not compress fragment blocks
119-noX do not compress extended attributes
120-no-fragments do not use fragments
121-always-use-fragments use fragment blocks for files larger than block size
122-no-duplicates do not perform duplicate checking
123-all-root make all files owned by root
124-force-uid uid set all file uids to uid
125-force-gid gid set all file gids to gid
126-nopad do not pad filesystem to a multiple of 4K
127-keep-as-directory if one source directory is specified, create a root
128 directory containing that directory, rather than the
129 contents of the directory
130
131Filesystem filter options:
132-p <pseudo-definition> Add pseudo file definition
133-pf <pseudo-file> Add list of pseudo file definitions
134-sort <sort_file> sort files according to priorities in <sort_file>. One
135 file or dir with priority per line. Priority -32768 to
136 32767, default priority 0
137-ef <exclude_file> list of exclude dirs/files. One per line
138-wildcards Allow extended shell wildcards (globbing) to be used in
139 exclude dirs/files
140-regex Allow POSIX regular expressions to be used in exclude
141 dirs/files
142
143Filesystem append options:
144-noappend do not append to existing filesystem
145-root-becomes <name> when appending source files/directories, make the
146 original root become a subdirectory in the new root
147 called <name>, rather than adding the new source items
148 to the original root
149
150Mksquashfs runtime options:
151-version print version, licence and copyright message
152-exit-on-error treat normally ignored errors as fatal
153-recover <name> recover filesystem data using recovery file <name>
154-no-recovery don't generate a recovery file
155-info print files written to filesystem
156-no-progress don't display the progress bar
157-progress display progress bar when using the -info option
158-processors <number> Use <number> processors. By default will use number of
159 processors available
160-mem <size> Use <size> physical memory. Currently set to 1922M
161 Optionally a suffix of K, M or G can be given to specify
162 Kbytes, Mbytes or Gbytes respectively
163
164Miscellaneous options:
165-root-owned alternative name for -all-root
166-noInodeCompression alternative name for -noI
167-noDataCompression alternative name for -noD
168-noFragmentCompression alternative name for -noF
169-noXattrCompression alternative name for -noX
170
171-Xhelp print compressor options for selected compressor
172
173Compressors available and compressor specific options:
174 gzip (default)
175 -Xcompression-level <compression-level>
176 <compression-level> should be 1 .. 9 (default 9)
177 -Xwindow-size <window-size>
178 <window-size> should be 8 .. 15 (default 15)
179 -Xstrategy strategy1,strategy2,...,strategyN
180 Compress using strategy1,strategy2,...,strategyN in turn
181 and choose the best compression.
182 Available strategies: default, filtered, huffman_only,
183 run_length_encoded and fixed
184 lzo
185 -Xalgorithm <algorithm>
186 Where <algorithm> is one of:
187 lzo1x_1
188 lzo1x_1_11
189 lzo1x_1_12
190 lzo1x_1_15
191 lzo1x_999 (default)
192 -Xcompression-level <compression-level>
193 <compression-level> should be 1 .. 9 (default 8)
194 Only applies to lzo1x_999 algorithm
195 lz4
196 -Xhc
197 Compress using LZ4 High Compression
198 xz
199 -Xbcj filter1,filter2,...,filterN
200 Compress using filter1,filter2,...,filterN in turn
201 (in addition to no filter), and choose the best compression.
202 Available filters: x86, arm, armthumb, powerpc, sparc, ia64
203 -Xdict-size <dict-size>
204 Use <dict-size> as the XZ dictionary size. The dictionary size
205 can be specified as a percentage of the block size, or as an
206 absolute value. The dictionary size must be less than or equal
207 to the block size and 8192 bytes or larger. It must also be
208 storable in the xz header as either 2^n or as 2^n+2^(n+1).
209 Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K
210 etc.
211
212Source1 source2 ... are the source directories/files containing the
213files/directories that will form the squashfs filesystem. If a single
214directory is specified (i.e. mksquashfs source output_fs) the squashfs
215filesystem will consist of that directory, with the top-level root
216directory corresponding to the source directory.
217
218If multiple source directories or files are specified, mksquashfs will merge
219the specified sources into a single filesystem, with the root directory
220containing each of the source files/directories. The name of each directory
221entry will be the basename of the source path. If more than one source
222entry maps to the same name, the conflicts are named xxx_1, xxx_2, etc. where
223xxx is the original name.
224
225To make this clear, take two example directories. Source directory
226"/home/phillip/test" contains "file1", "file2" and "dir1".
227Source directory "goodies" contains "goodies1", "goodies2" and "goodies3".
228
229usage example 1:
230
231%mksquashfs /home/phillip/test output_fs
232
233This will generate a squashfs filesystem with root entries
234"file1", "file2" and "dir1".
235
236example 2:
237
238%mksquashfs /home/phillip/test goodies output_fs
239
240This will create a squashfs filesystem with the root containing
241entries "test" and "goodies" corresponding to the source
242directories "/home/phillip/test" and "goodies".
243
244example 3:
245
246%mksquashfs /home/phillip/test goodies test output_fs
247
248This is the same as the previous example, except a third
249source directory "test" has been specified. This conflicts
250with the first directory named "test" and will be renamed "test_1".
251
252Multiple sources allow filesystems to be generated without needing to
253copy all source files into a common directory. This simplifies creating
254filesystems.
255
256The -keep-as-directory option can be used when only one source directory
257is specified, and you wish the root to contain that directory, rather than
258the contents of the directory. For example:
259
260example 4:
261
262%mksquashfs /home/phillip/test output_fs -keep-as-directory
263
264This is the same as example 1, except for -keep-as-directory.
265This will generate a root directory containing directory "test",
266rather than the "test" directory contents "file1", "file2" and "dir1".
267
268The Dest argument is the destination where the squashfs filesystem will be
269written. This can either be a conventional file or a block device. If the file
270doesn't exist it will be created, if it does exist and a squashfs
271filesystem exists on it, mksquashfs will append. The -noappend option will
272write a new filesystem irrespective of whether an existing filesystem is
273present.
274
2753.2 Changing compression algorithm and compression specific options
276-------------------------------------------------------------------
277
278By default Mksquashfs will compress using the gzip compression
279algorithm. This algorithm offers a good trade-off between compression
280ratio, and memory and time taken to decompress.
281
282Squashfs also supports LZ4, LZO and XZ (LZMA2) compression. LZO offers worse
283compression ratio than gzip, but is faster to decompress. XZ offers better
284compression ratio than gzip, but at the expense of greater memory and time
285to decompress (and significantly more time to compress). LZ4 is similar
286to LZO, but, support for it is not yet in the mainline kernel, and so
287its usefulness is currently limited to using Squashfs with Mksquashfs/Unsquashfs
288as an archival system like tar.
289
290If you're not building the squashfs-tools and kernel from source, then
291the tools and kernel may or may not have been built with support for LZ4, LZO or
292XZ compression. The compression algorithms supported by the build of
293Mksquashfs can be found by typing mksquashfs without any arguments. The
294compressors available are displayed at the end of the help message, e.g.
295
296Compressors available and compressor specific options:
297 gzip (default)
298 -Xcompression-level <compression-level>
299 <compression-level> should be 1 .. 9 (default 9)
300 -Xwindow-size <window-size>
301 <window-size> should be 8 .. 15 (default 15)
302 -Xstrategy strategy1,strategy2,...,strategyN
303 Compress using strategy1,strategy2,...,strategyN in turn
304 and choose the best compression.
305 Available strategies: default, filtered, huffman_only,
306 run_length_encoded and fixed
307 lzo
308 -Xalgorithm <algorithm>
309 Where <algorithm> is one of:
310 lzo1x_1
311 lzo1x_1_11
312 lzo1x_1_12
313 lzo1x_1_15
314 lzo1x_999 (default)
315 -Xcompression-level <compression-level>
316 <compression-level> should be 1 .. 9 (default 8)
317 Only applies to lzo1x_999 algorithm
318 lz4
319 -Xhc
320 Compress using LZ4 High Compression
321 xz
322 -Xbcj filter1,filter2,...,filterN
323 Compress using filter1,filter2,...,filterN in turn
324 (in addition to no filter), and choose the best compression.
325 Available filters: x86, arm, armthumb, powerpc, sparc, ia64
326 -Xdict-size <dict-size>
327 Use <dict-size> as the XZ dictionary size. The dictionary size
328 can be specified as a percentage of the block size, or as an
329 absolute value. The dictionary size must be less than or equal
330 to the block size and 8192 bytes or larger. It must also be
331 storable in the xz header as either 2^n or as 2^n+2^(n+1).
332 Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K
333 etc.
334
335If the compressor offers compression specific options (all the compressors now
336have compression specific options except the deprecated lzma1 compressor)
337then these options are also displayed (.i.e. in the above XZ is shown with two
338compression specific options). The compression specific options are, obviously,
339specific to the compressor in question, and the compressor documentation and
340web sites should be consulted to understand their behaviour. In general
341the Mksquashfs compression defaults for each compressor are optimised to
342give the best performance for each compressor, where what constitutes
343best depends on the compressor. For gzip/xz best means highest compression,
344for LZO/LZ4 best means a tradeoff between compression and (de)-compression
345overhead (LZO/LZ4 by definition are intended for weaker processors).
346
3473.3 Changing global compression defaults used in mksquashfs
348-----------------------------------------------------------
349
350There are a large number of options that can be used to control the
351compression in mksquashfs. By and large the defaults are the most
352optimum settings and should only be changed in exceptional circumstances!
353Note, this does not apply to the block size, increasing the block size
354from the default of 128Kbytes will increase compression (especially
355for the xz compressor) and should increase I/O performance too. However,
356a block size of greater than 128Kbytes may increase latency in certain
357cases (where the filesystem contains lots of fragments, and no locality
358of reference is observed). For this reason the block size default is
359configured to the less optimal 128Kbytes. Users should experiment
360with 256Kbyte sizes or above.
361
362The -noI, -noD and -noF options (also -noInodeCompression, -noDataCompression
363and -noFragmentCompression) can be used to force mksquashfs to not compress
364inodes/directories, data and fragments respectively. Giving all options
365generates an uncompressed filesystem.
366
367The -no-fragments tells mksquashfs to not generate fragment blocks, and rather
368generate a filesystem similar to a Squashfs 1.x filesystem. It will of course
369still be a Squashfs 4.0 filesystem but without fragments, and so it won't be
370mountable on a Squashfs 1.x system.
371
372The -always-use-fragments option tells mksquashfs to always generate
373fragments for files irrespective of the file length. By default only small
374files less than the block size are packed into fragment blocks. The ends of
375files which do not fit fully into a block, are NOT by default packed into
376fragments. To illustrate this, a 100K file has an initial 64K block and a 36K
377remainder. This 36K remainder is not packed into a fragment by default. This
378is because to do so leads to a 10 - 20% drop in sequential I/O performance, as a
379disk head seek is needed to seek to the initial file data and another disk seek
380is need to seek to the fragment block. Specify this option if you want file
381remainders to be packed into fragment blocks. Doing so may increase the
382compression obtained BUT at the expense of I/O speed.
383
384The -no-duplicates option tells mksquashfs to not check the files being
385added to the filesystem for duplicates. This can result in quicker filesystem
386generation and appending although obviously compression will suffer badly if
387there is a lot of duplicate files.
388
389The -b option allows the block size to be selected, both "K" and "M" postfixes
390are supported, this can be either 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K or
3911M bytes.
392
3933.4 Specifying the UIDs/GIDs used in the filesystem
394---------------------------------------------------
395
396By default files in the generated filesystem inherit the UID and GID ownership
397of the original file. However, mksquashfs provides a number of options which
398can be used to override the ownership.
399
400The options -all-root and -root-owned (both do exactly the same thing) force all
401file uids/gids in the generated Squashfs filesystem to be root. This allows
402root owned filesystems to be built without root access on the host machine.
403
404The "-force-uid uid" option forces all files in the generated Squashfs
405filesystem to be owned by the specified uid. The uid can be specified either by
406name (i.e. "root") or by number.
407
408The "-force-gid gid" option forces all files in the generated Squashfs
409filesystem to be group owned by the specified gid. The gid can be specified
410either by name (i.e. "root") or by number.
411
4123.5 Excluding files from the filesystem
413---------------------------------------
414
415The -e and -ef options allow files/directories to be specified which are
416excluded from the output filesystem. The -e option takes the exclude
417files/directories from the command line, the -ef option takes the
418exlude files/directories from the specified exclude file, one file/directory
419per line.
420
421Two styles of exclude file matching are supported: basic exclude matching, and
422extended wildcard matching. Basic exclude matching is a legacy feature
423retained for backwards compatibility with earlier versions of Mksquashfs.
424Extended wildcard matching should be used in preference.
425
4263.5.1 Basic exclude matching
427----------------------------
428
429Each exclude file is treated as an exact match of a file/directory in
430the source directories. If an exclude file/directory is absolute (i.e.
431prefixed with /, ../, or ./) the entry is treated as absolute, however, if an
432exclude file/directory is relative, it is treated as being relative to each of
433the sources in turn, i.e.
434
435%mksquashfs /tmp/source1 source2 output_fs -e ex1 /tmp/source1/ex2 out/ex3
436
437Will generate exclude files /tmp/source1/ex2, /tmp/source1/ex1, source2/ex1,
438/tmp/source1/out/ex3 and source2/out/ex3.
439
4403.5.2 Extended exclude file handling
441------------------------------------
442
443Extended exclude file matching treats each exclude file as a wildcard or
444regex expression. To enable wildcard matching specify the -wildcards
445option, and to enable regex matching specify the -regex option. In most
446cases the -wildcards option should be used rather than -regex because wildcard
447matching behaviour is significantly easier to understand!
448
449In addition to wildcards/regex expressions, exclude files can be "anchored" or
450"non-anchored". An anchored exclude is one which matches from the root of the
451directory and nowhere else, a non-anchored exclude matches anywhere. For
452example given the directory hierarchy "a/b/c/a/b", the anchored exclude
453"a/b" will match "a/b" at the root of the directory hierarchy, but
454it will not match the "/a/b" sub-directory within directory "c", whereas a
455non-anchored exclude would.
456
457A couple of examples should make this clearer.
458
459Anchored excludes
460
461 1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz'
462
463 Exclude all files matching "*.gz" in the top level directory "test".
464
465 2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*'
466
467 Exclude all files beginning with "example" inside directories called
468 "Test" or "test", that occur inside any top level directory.
469
470 Using extended wildcards, negative matching is also possible.
471
472 3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz'
473
474 Exclude all files matching "*.gz" in top level directory "test",
475 except those with "data" in the name.
476
477Non-anchored excludes
478
479 By default excludes match from the top level directory, but it is
480 often useful to exclude a file matching anywhere in the source directories.
481 For this non-anchored excludes can be used, specified by pre-fixing the
482 exclude with "...".
483
484 Examples:
485
486 1. mksquashfs example image.sqsh -wildcards -e '... *.gz'
487
488 Exclude files matching "*.gz" anywhere in the source directories.
489 For example this will match "example.gz", "test/example.gz", and
490 "test/test/example.gz".
491
492 2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz'
493
494 Exclude files matching "*.gz" inside directories called "Test" or
495 "test" that occur anywhere in the source directories.
496
497 Again, using extended wildcards, negative matching is also possible.
498
499 3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz'
500
501 Exclude all files matching "*.gz" anywhere in the source directories,
502 except those with "data" in the name.
503
5043.5.3 Exclude files summary
505---------------------------
506
507The -e and -ef exclude options are usefully used in archiving the entire
508filesystem, where it is wished to avoid archiving /proc, and the filesystem
509being generated, i.e.
510
511%mksquashfs / /tmp/root.sqsh -e proc /tmp/root.sqsh
512
513Multiple -ef options can be specified on the command line, and the -ef
514option can be used in conjuction with the -e option.
515
5163.6 Appending to squashfs filesystems
517-------------------------------------
518
519Running squashfs with the destination directory containing an existing
520filesystem will add the source items to the existing filesystem. By default,
521the source items are added to the existing root directory.
522
523To make this clear... An existing filesystem "image" contains root entries
524"old1", and "old2". Source directory "/home/phillip/test" contains "file1",
525"file2" and "dir1".
526
527example 1:
528
529%mksquashfs /home/phillip/test image
530
531Will create a new "image" with root entries "old1", "old2", "file1", "file2" and
532"dir1"
533
534example 2:
535
536%mksquashfs /home/phillip/test image -keep-as-directory
537
538Will create a new "image" with root entries "old1", "old2", and "test".
539As shown in the previous section, for single source directories
540'-keep-as-directory' adds the source directory rather than the
541contents of the directory.
542
543example 3:
544
545%mksquashfs /home/phillip/test image -keep-as-directory -root-becomes
546original-root
547
548Will create a new "image" with root entries "original-root", and "test". The
549'-root-becomes' option specifies that the original root becomes a subdirectory
550in the new root, with the specified name.
551
552The append option with file duplicate detection, means squashfs can be
553used as a simple versioning archiving filesystem. A squashfs filesystem can
554be created with for example the linux-2.4.19 source. Appending the linux-2.4.20
555source will create a filesystem with the two source trees, but only the
556changed files will take extra room, the unchanged files will be detected as
557duplicates.
558
5593.7 Appending recovery file feature
560-----------------------------------
561
562Recovery files are created when appending to existing Squashfs
563filesystems. This allows the original filesystem to be recovered
564if Mksquashfs aborts unexpectedly (i.e. power failure).
565
566The recovery files are called squashfs_recovery_xxx_yyy, where
567"xxx" is the name of the filesystem being appended to, and "yyy" is a
568number to guarantee filename uniqueness (the PID of the parent Mksquashfs
569process).
570
571Normally if Mksquashfs exits correctly the recovery file is deleted to
572avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover"
573option can be used to recover the filesystem, giving the previously
574created recovery file as a parameter, i.e.
575
576mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234
577
578The writing of the recovery file can be disabled by specifying the
579"-no-recovery" option.
580
5813.8 Pseudo file support
582-----------------------
583
584Mksquashfs supports pseudo files, these allow fake files, directories, character
585and block devices to be specified and added to the Squashfs filesystem being
586built, rather than requiring them to be present in the source directories.
587This, for example, allows device nodes to be added to the filesystem without
588requiring root access.
589
590Mksquashfs 4.1 added support for "dynamic pseudo files" and a modify operation.
591Dynamic pseudo files allow files to be dynamically created when Mksquashfs
592is run, their contents being the result of running a command or piece of
593shell script. The modifiy operation allows the mode/uid/gid of an existing
594file in the source filesystem to be modified.
595
596Two Mksquashfs options are supported, -p allows one pseudo file to be specified
597on the command line, and -pf allows a pseudo file to be specified containing a
598list of pseduo definitions, one per line.
599
6003.8.1. Creating a dynamic file
601------------------------------
602
603Pseudo definition
604
605Filename f mode uid gid command
606
607mode is the octal mode specifier, similar to that expected by chmod.
608
609uid and gid can be either specified as a decimal number, or by name.
610
611command can be an executable or a piece of shell script, and it is executed
612by running "/bin/sh -c command". The stdout becomes the contents of
613"Filename".
614
615Examples:
616
617Running a basic command
618-----------------------
619
620/somedir/dmesg f 444 root root dmesg
621
622creates a file "/somedir/dmesg" containing the output from dmesg.
623
624Executing shell script
625----------------------
626
627RELEASE f 444 root root \
628 if [ ! -e /tmp/ver ]; then \
629 echo 0 > /tmp/ver; \
630 fi; \
631 ver=`cat /tmp/ver`; \
632 ver=$((ver +1)); \
633 echo $ver > /tmp/ver; \
634 echo -n `cat /tmp/release`; \
635 echo "-dev #"$ver `date` "Build host" `hostname`
636
637Creates a file RELEASE containing the release name, date, build host, and
638an incrementing version number. The incrementing version is a side-effect
639of executing the shell script, and ensures every time Mksquashfs is run a
640new version number is used without requiring any other shell scripting.
641
642The above example also shows that commands can be split across multiple lines
643using "\". Obviously as the script will be presented to the shell as a single
644line, a semicolon is need to separate individual shell commands within the
645shell script.
646
647Reading from a device (or fifo/named socket)
648--------------------------------------------
649
650input f 444 root root dd if=/dev/sda1 bs=1024 count=10
651
652Copies 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs
653given a device, fifo, or named socket will place that special file within the
654Squashfs filesystem, the above allows input from these special files to be
655captured and placed in the Squashfs filesystem.
656
6573.8.2. Creating a block or character device
658-------------------------------------------
659
660Pseudo definition
661
662Filename type mode uid gid major minor
663
664Where type is either
665 b - for block devices, and
666 c - for character devices
667
668mode is the octal mode specifier, similar to that expected by chmod.
669
670uid and gid can be either specified as a decimal number, or by name.
671
672For example:
673
674/dev/chr_dev c 666 root root 100 1
675/dev/blk_dev b 666 0 0 200 200
676
677creates a character device "/dev/chr_dev" with major:minor 100:1 and
678a block device "/dev/blk_dev" with major:minor 200:200, both with root
679uid/gid and a mode of rw-rw-rw.
680
6813.8.3. Creating a directory
682---------------------------
683
684Pseudo definition
685
686Filename d mode uid gid
687
688mode is the octal mode specifier, similar to that expected by chmod.
689
690uid and gid can be either specified as a decimal number, or by name.
691
692For example:
693
694/pseudo_dir d 666 root root
695
696creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw.
697
6983.8.4. Modifying attributes of an existing file
699-----------------------------------------------
700
701Pseudo definition
702
703Filename m mode uid gid
704
705mode is the octal mode specifier, similar to that expected by chmod.
706
707uid and gid can be either specified as a decimal number, or by name.
708
709For example:
710
711dmesg m 666 root root
712
713Changes the attributes of the file "dmesg" in the filesystem to have
714root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained
715from the source filesystem.
716
7173.9 Miscellaneous options
718-------------------------
719
720The -info option displays the files/directories as they are compressed and
721added to the filesystem. The original uncompressed size of each file
722is printed, along with DUPLICATE if the file is a duplicate of a
723file in the filesystem.
724
725The -nopad option informs mksquashfs to not pad the filesystem to a 4K multiple.
726This is performed by default to enable the output filesystem file to be mounted
727by loopback, which requires files to be a 4K multiple. If the filesystem is
728being written to a block device, or is to be stored in a bootimage, the extra
729pad bytes are not needed.
730
7314. UNSQUASHFS
732-------------
733
734Unsquashfs allows you to decompress and extract a Squashfs filesystem without
735mounting it. It can extract the entire filesystem, or a specific
736file or directory.
737
738The Unsquashfs usage info is:
739
740SYNTAX: ./unsquashfs [options] filesystem [directories or files to extract]
741 -v[ersion] print version, licence and copyright information
742 -d[est] <pathname> unsquash to <pathname>, default "squashfs-root"
743 -n[o-progress] don't display the progress bar
744 -no[-xattrs] don't extract xattrs in file system
745 -x[attrs] extract xattrs in file system (default)
746 -u[ser-xattrs] only extract user xattrs in file system.
747 Enables extracting xattrs
748 -p[rocessors] <number> use <number> processors. By default will use
749 number of processors available
750 -i[nfo] print files as they are unsquashed
751 -li[nfo] print files as they are unsquashed with file
752 attributes (like ls -l output)
753 -l[s] list filesystem, but don't unsquash
754 -ll[s] list filesystem with file attributes (like
755 ls -l output), but don't unsquash
756 -f[orce] if file already exists then overwrite
757 -s[tat] display filesystem superblock information
758 -e[f] <extract file> list of directories or files to extract.
759 One per line
760 -da[ta-queue] <size> Set data queue to <size> Mbytes. Default 256
761 Mbytes
762 -fr[ag-queue] <size> Set fragment queue to <size> Mbytes. Default
763 256 Mbytes
764 -r[egex] treat extract names as POSIX regular expressions
765 rather than use the default shell wildcard
766 expansion (globbing)
767
768Decompressors available:
769 gzip
770 lzo
771 lz4
772 xz
773
774To extract a subset of the filesystem, the filenames or directory
775trees that are to be extracted can be specified on the command line. The
776files/directories should be specified using the full path to the
777files/directories as they appear within the Squashfs filesystem. The
778files/directories will also be extracted to those positions within the specified
779destination directory.
780
781The extract files can also be given in a file using the "-e[f]" option.
782
783Similarly to Mksquashfs, wildcard matching is performed on the extract
784files. Wildcard matching is enabled by default.
785
786Examples:
787
788 1. unsquashfs image.sqsh 'test/*.gz'
789
790 Extract all files matching "*.gz" in the top level directory "test".
791
792 2. unsquashfs image.sqsh '[Tt]est/example*'
793
794 Extract all files beginning with "example" inside top level directories
795 called "Test" or "test".
796
797 Using extended wildcards, negative matching is also possible.
798
799 3. unsquashfs image.sqsh 'test/!(*data*).gz'
800
801 Extract all files matching "*.gz" in top level directory "test",
802 except those with "data" in the name.
803
804
8054.1 Unsquashfs options
806----------------------
807
808The "-ls" option can be used to list the contents of a filesystem without
809decompressing the filesystem data itself. The "-lls" option is similar
810but it also displays file attributes (ls -l style output).
811
812The "-info" option forces Unsquashfs to print each file as it is decompressed.
813The -"linfo" is similar but it also displays file attributes.
814
815The "-dest" option specifies the directory that is used to decompress
816the filesystem data. If this option is not given then the filesystem is
817decompressed to the directory "squashfs-root" in the current working directory.
818
819The "-force" option forces Unsquashfs to output to the destination
820directory even if files or directories already exist. This allows you
821to update an existing directory tree, or to Unsquashfs to a partially
822filled directory. Without the "-force" option, Unsquashfs will
823refuse to overwrite any existing files, or to create any directories if they
824already exist. This is done to protect data in case of mistakes, and
825so the "-force" option should be used with caution.
826
827The "-stat" option displays filesystem superblock information. This is
828useful to discover the filesystem version, byte ordering, whether it has a NFS
829export table, and what options were used to compress the filesystem, etc.
830
831Unsquashfs can decompress all Squashfs filesystem versions, 1.x, 2.x, 3.x and
8324.0 filesystems.
833
8345. FILESYSTEM LAYOUT
835--------------------
836
837A squashfs filesystem consists of a maximum of nine parts, packed together on a
838byte alignment:
839
840 ---------------
841 | superblock |
842 |---------------|
843 | compression |
844 | options |
845 |---------------|
846 | datablocks |
847 | & fragments |
848 |---------------|
849 | inode table |
850 |---------------|
851 | directory |
852 | table |
853 |---------------|
854 | fragment |
855 | table |
856 |---------------|
857 | export |
858 | table |
859 |---------------|
860 | uid/gid |
861 | lookup table |
862 |---------------|
863 | xattr |
864 | table |
865 ---------------
866
867Compressed data blocks are written to the filesystem as files are read from
868the source directory, and checked for duplicates. Once all file data has been
869written the completed super-block, compression options, inode, directory,
870fragment, export, uid/gid lookup and xattr tables are written.
871
8725.1 Compression options
873-----------------------
874
875Compressors can optionally support compression specific options (e.g.
876dictionary size). If non-default compression options have been used, then
877these are stored here.
878
8795.2 Inodes
880----------
881
882Metadata (inodes and directories) are compressed in 8Kbyte blocks. Each
883compressed block is prefixed by a two byte length, the top bit is set if the
884block is uncompressed. A block will be uncompressed if the -noI option is set,
885or if the compressed block was larger than the uncompressed block.
886
887Inodes are packed into the metadata blocks, and are not aligned to block
888boundaries, therefore inodes overlap compressed blocks. Inodes are identified
889by a 48-bit number which encodes the location of the compressed metadata block
890containing the inode, and the byte offset into that block where the inode is
891placed (<block, offset>).
892
893To maximise compression there are different inodes for each file type
894(regular file, directory, device, etc.), the inode contents and length
895varying with the type.
896
897To further maximise compression, two types of regular file inode and
898directory inode are defined: inodes optimised for frequently occurring
899regular files and directories, and extended types where extra
900information has to be stored.
901
9025.3 Directories
903---------------
904
905Like inodes, directories are packed into compressed metadata blocks, stored
906in a directory table. Directories are accessed using the start address of
907the metablock containing the directory and the offset into the
908decompressed block (<block, offset>).
909
910Directories are organised in a slightly complex way, and are not simply
911a list of file names. The organisation takes advantage of the
912fact that (in most cases) the inodes of the files will be in the same
913compressed metadata block, and therefore, can share the start block.
914Directories are therefore organised in a two level list, a directory
915header containing the shared start block value, and a sequence of directory
916entries, each of which share the shared start block. A new directory header
917is written once/if the inode start block changes. The directory
918header/directory entry list is repeated as many times as necessary.
919
920Directories are sorted, and can contain a directory index to speed up
921file lookup. Directory indexes store one entry per metablock, each entry
922storing the index/filename mapping to the first directory header
923in each metadata block. Directories are sorted in alphabetical order,
924and at lookup the index is scanned linearly looking for the first filename
925alphabetically larger than the filename being looked up. At this point the
926location of the metadata block the filename is in has been found.
927The general idea of the index is ensure only one metadata block needs to be
928decompressed to do a lookup irrespective of the length of the directory.
929This scheme has the advantage that it doesn't require extra memory overhead
930and doesn't require much extra storage on disk.
931
9325.4 File data
933-------------
934
935Regular files consist of a sequence of contiguous compressed blocks, and/or a
936compressed fragment block (tail-end packed block). The compressed size
937of each datablock is stored in a block list contained within the
938file inode.
939
940To speed up access to datablocks when reading 'large' files (256 Mbytes or
941larger), the code implements an index cache that caches the mapping from
942block index to datablock location on disk.
943
944The index cache allows Squashfs to handle large files (up to 1.75 TiB) while
945retaining a simple and space-efficient block list on disk. The cache
946is split into slots, caching up to eight 224 GiB files (128 KiB blocks).
947Larger files use multiple slots, with 1.75 TiB files using all 8 slots.
948The index cache is designed to be memory efficient, and by default uses
94916 KiB.
950
9515.5 Fragment lookup table
952-------------------------
953
954Regular files can contain a fragment index which is mapped to a fragment
955location on disk and compressed size using a fragment lookup table. This
956fragment lookup table is itself stored compressed into metadata blocks.
957A second index table is used to locate these. This second index table for
958speed of access (and because it is small) is read at mount time and cached
959in memory.
960
9615.6 Uid/gid lookup table
962------------------------
963
964For space efficiency regular files store uid and gid indexes, which are
965converted to 32-bit uids/gids using an id look up table. This table is
966stored compressed into metadata blocks. A second index table is used to
967locate these. This second index table for speed of access (and because it
968is small) is read at mount time and cached in memory.
969
9705.7 Export table
971----------------
972
973To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems
974can optionally (disabled with the -no-exports Mksquashfs option) contain
975an inode number to inode disk location lookup table. This is required to
976enable Squashfs to map inode numbers passed in filehandles to the inode
977location on disk, which is necessary when the export code reinstantiates
978expired/flushed inodes.
979
980This table is stored compressed into metadata blocks. A second index table is
981used to locate these. This second index table for speed of access (and because
982it is small) is read at mount time and cached in memory.
983
9845.8 Xattr table
985---------------
986
987The xattr table contains extended attributes for each inode. The xattrs
988for each inode are stored in a list, each list entry containing a type,
989name and value field. The type field encodes the xattr prefix
990("user.", "trusted." etc) and it also encodes how the name/value fields
991should be interpreted. Currently the type indicates whether the value
992is stored inline (in which case the value field contains the xattr value),
993or if it is stored out of line (in which case the value field stores a
994reference to where the actual value is stored). This allows large values
995to be stored out of line improving scanning and lookup performance and it
996also allows values to be de-duplicated, the value being stored once, and
997all other occurences holding an out of line reference to that value.
998
999The xattr lists are packed into compressed 8K metadata blocks.
1000To reduce overhead in inodes, rather than storing the on-disk
1001location of the xattr list inside each inode, a 32-bit xattr id
1002is stored. This xattr id is mapped into the location of the xattr
1003list using a second xattr id lookup table.
1004
10056. AUTHOR INFO
1006--------------
1007
1008Squashfs was written by Phillip Lougher, email phillip@lougher.demon.co.uk,
1009in Chepstow, Wales, UK. If you like the program, or have any problems,
1010then please email me, as it's nice to get feedback!
1011