19e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project/* 29e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project * gzlog.c 304351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes * Copyright (C) 2004, 2008, 2012 Mark Adler, all rights reserved 49e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project * For conditions of distribution and use, see copyright notice in gzlog.h 504351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes * version 2.2, 14 Aug 2012 69e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project */ 79e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 8381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* 9381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes gzlog provides a mechanism for frequently appending short strings to a gzip 10381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes file that is efficient both in execution time and compression ratio. The 11381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strategy is to write the short strings in an uncompressed form to the end of 12381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the gzip file, only compressing when the amount of uncompressed data has 13381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes reached a given threshold. 14381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 15381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes gzlog also provides protection against interruptions in the process due to 16381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes system crashes. The status of the operation is recorded in an extra field 17381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes in the gzip file, and is only updated once the gzip file is brought to a 18381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes valid state. The last data to be appended or compressed is saved in an 19381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes auxiliary file, so that if the operation is interrupted, it can be completed 20381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the next time an append operation is attempted. 21381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 22381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes gzlog maintains another auxiliary file with the last 32K of data from the 23381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes compressed portion, which is preloaded for the compression of the subsequent 24381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes data. This minimizes the impact to the compression ratio of appending. 25381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes */ 26381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 27381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* 28381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Operations Concept: 29381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 30381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Files (log name "foo"): 31381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.gz -- gzip file with the complete log 32381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.add -- last message to append or last data to compress 33381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.dict -- dictionary of the last 32K of data for next compression 34381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.temp -- temporary dictionary file for compression after this one 35381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.lock -- lock file for reading and writing the other files 36381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.repairs -- log file for log file recovery operations (not compressed) 37381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 38381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes gzip file structure: 39381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - fixed-length (no file name) header with extra field (see below) 40381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - compressed data ending initially with empty stored block 41381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - uncompressed data filling out originally empty stored block and 42381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes subsequent stored blocks as needed (16K max each) 43381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - gzip trailer 44381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - no junk at end (no other gzip streams) 45381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 46381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes When appending data, the information in the first three items above plus the 47381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.add file are sufficient to recover an interrupted append operation. The 48381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes extra field has the necessary information to restore the start of the last 49381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes stored block and determine where to append the data in the foo.add file, as 50381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes well as the crc and length of the gzip data before the append operation. 51381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 52381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes The foo.add file is created before the gzip file is marked for append, and 53381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes deleted after the gzip file is marked as complete. So if the append 54381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes operation is interrupted, the data to add will still be there. If due to 55381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes some external force, the foo.add file gets deleted between when the append 56381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes operation was interrupted and when recovery is attempted, the gzip file will 57381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes still be restored, but without the appended data. 58381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 59381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes When compressing data, the information in the first two items above plus the 60381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.add file are sufficient to recover an interrupted compress operation. 61381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes The extra field has the necessary information to find the end of the 62381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes compressed data, and contains both the crc and length of just the compressed 63381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes data and of the complete set of data including the contents of the foo.add 64381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes file. 65381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 66381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Again, the foo.add file is maintained during the compress operation in case 67381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes of an interruption. If in the unlikely event the foo.add file with the data 68381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes to be compressed is missing due to some external force, a gzip file with 69381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes just the previous compressed data will be reconstructed. In this case, all 70381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes of the data that was to be compressed is lost (approximately one megabyte). 71381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes This will not occur if all that happened was an interruption of the compress 72381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes operation. 73381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 74381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes The third state that is marked is the replacement of the old dictionary with 75381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the new dictionary after a compress operation. Once compression is 76381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes complete, the gzip file is marked as being in the replace state. This 77381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes completes the gzip file, so an interrupt after being so marked does not 78381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes result in recompression. Then the dictionary file is replaced, and the gzip 79381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes file is marked as completed. This state prevents the possibility of 80381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes restarting compression with the wrong dictionary file. 81381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 82381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes All three operations are wrapped by a lock/unlock procedure. In order to 83381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes gain exclusive access to the log files, first a foo.lock file must be 84381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes exclusively created. When all operations are complete, the lock is 85381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes released by deleting the foo.lock file. If when attempting to create the 86381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes lock file, it already exists and the modify time of the lock file is more 87381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes than five minutes old (set by the PATIENCE define below), then the old 88381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes lock file is considered stale and deleted, and the exclusive creation of 89381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the lock file is retried. To assure that there are no false assessments 90381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes of the staleness of the lock file, the operations periodically touch the 91381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes lock file to update the modified date. 92381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 93381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Following is the definition of the extra field with all of the information 94381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes required to enable the above append and compress operations and their 95381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes recovery if interrupted. Multi-byte values are stored little endian 96381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes (consistent with the gzip format). File pointers are eight bytes long. 97381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes The crc's and lengths for the gzip trailer are four bytes long. (Note that 98381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the length at the end of a gzip file is used for error checking only, and 99381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes for large files is actually the length modulo 2^32.) The stored block 100381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes length is two bytes long. The gzip extra field two-byte identification is 101381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes "ap" for append. It is assumed that writing the extra field to the file is 102381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes an "atomic" operation. That is, either all of the extra field is written 103381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes to the file, or none of it is, if the operation is interrupted right at the 104381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes point of updating the extra field. This is a reasonable assumption, since 105381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the extra field is within the first 52 bytes of the file, which is smaller 106381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes than any expected block size for a mass storage device (usually 512 bytes or 107381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes larger). 108381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 109381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Extra field (35 bytes): 110381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Pointer to first stored block length -- this points to the two-byte length 111381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes of the first stored block, which is followed by the two-byte, one's 112381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes complement of that length. The stored block length is preceded by the 113381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes three-bit header of the stored block, which is the actual start of the 114381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes stored block in the deflate format. See the bit offset field below. 115381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Pointer to the last stored block length. This is the same as above, but 116381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes for the last stored block of the uncompressed data in the gzip file. 117381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Initially this is the same as the first stored block length pointer. 118381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes When the stored block gets to 16K (see the MAX_STORE define), then a new 119381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes stored block as added, at which point the last stored block length pointer 120381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes is different from the first stored block length pointer. When they are 121381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes different, the first bit of the last stored block header is eight bits, or 122381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes one byte back from the block length. 123381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Compressed data crc and length. This is the crc and length of the data 124381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes that is in the compressed portion of the deflate stream. These are used 125381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes only in the event that the foo.add file containing the data to compress is 126381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes lost after a compress operation is interrupted. 127381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Total data crc and length. This is the crc and length of all of the data 128381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes stored in the gzip file, compressed and uncompressed. It is used to 129381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes reconstruct the gzip trailer when compressing, as well as when recovering 130381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes interrupted operations. 131381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Final stored block length. This is used to quickly find where to append, 132381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes and allows the restoration of the original final stored block state when 133381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes an append operation is interrupted. 134381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - First stored block start as the number of bits back from the final stored 135381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes block first length byte. This value is in the range of 3..10, and is 136381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes stored as the low three bits of the final byte of the extra field after 137381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes subtracting three (0..7). This allows the last-block bit of the stored 138381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes block header to be updated when a new stored block is added, for the case 139381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes when the first stored block and the last stored block are the same. (When 140381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes they are different, the numbers of bits back is known to be eight.) This 141381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes also allows for new compressed data to be appended to the old compressed 142381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes data in the compress operation, overwriting the previous first stored 143381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes block, or for the compressed data to be terminated and a valid gzip file 144381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes reconstructed on the off chance that a compression operation was 145381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes interrupted and the data to compress in the foo.add file was deleted. 146381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - The operation in process. This is the next two bits in the last byte (the 147381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes bits under the mask 0x18). The are interpreted as 0: nothing in process, 148381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 1: append in process, 2: compress in process, 3: replace in process. 149381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - The top three bits of the last byte in the extra field are reserved and 150381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes are currently set to zero. 151381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 152381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Main procedure: 153381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Exclusively create the foo.lock file using the O_CREAT and O_EXCL modes of 154381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the system open() call. If the modify time of an existing lock file is 155381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes more than PATIENCE seconds old, then the lock file is deleted and the 156381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes exclusive create is retried. 157381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Load the extra field from the foo.gz file, and see if an operation was in 158381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes progress but not completed. If so, apply the recovery procedure below. 159381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Perform the append procedure with the provided data. 160381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - If the uncompressed data in the foo.gz file is 1MB or more, apply the 161381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes compress procedure. 162381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Delete the foo.lock file. 163381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 164381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Append procedure: 165381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Put what to append in the foo.add file so that the operation can be 166381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes restarted if this procedure is interrupted. 167381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Mark the foo.gz extra field with the append operation in progress. 168381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes + Restore the original last-block bit and stored block length of the last 169381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes stored block from the information in the extra field, in case a previous 170381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes append operation was interrupted. 171381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Append the provided data to the last stored block, creating new stored 172381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes blocks as needed and updating the stored blocks last-block bits and 173381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes lengths. 174381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Update the crc and length with the new data, and write the gzip trailer. 175381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Write over the extra field (with a single write operation) with the new 176381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes pointers, lengths, and crc's, and mark the gzip file as not in process. 177381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Though there is still a foo.add file, it will be ignored since nothing 178381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes is in process. If a foo.add file is leftover from a previously 179381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes completed operation, it is truncated when writing new data to it. 180381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Delete the foo.add file. 181381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 182381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Compress and replace procedures: 183381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Read all of the uncompressed data in the stored blocks in foo.gz and write 184381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes it to foo.add. Also write foo.temp with the last 32K of that data to 185381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes provide a dictionary for the next invocation of this procedure. 186381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Rewrite the extra field marking foo.gz with a compression in process. 187381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes * If there is no data provided to compress (due to a missing foo.add file 188381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes when recovering), reconstruct and truncate the foo.gz file to contain 189381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes only the previous compressed data and proceed to the step after the next 190381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes one. Otherwise ... 191381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Compress the data with the dictionary in foo.dict, and write to the 192381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.gz file starting at the bit immediately following the last previously 193381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes compressed block. If there is no foo.dict, proceed anyway with the 194381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes compression at slightly reduced efficiency. (For the foo.dict file to be 195381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes missing requires some external failure beyond simply the interruption of 196381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes a compress operation.) During this process, the foo.lock file is 197381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes periodically touched to assure that that file is not considered stale by 198381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes another process before we're done. The deflation is terminated with a 199381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes non-last empty static block (10 bits long), that is then located and 200381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes written over by a last-bit-set empty stored block. 201381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Append the crc and length of the data in the gzip file (previously 202381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes calculated during the append operations). 203381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Write over the extra field with the updated stored block offsets, bits 204381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes back, crc's, and lengths, and mark foo.gz as in process for a replacement 205381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes of the dictionary. 206381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes @ Delete the foo.add file. 207381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Replace foo.dict with foo.temp. 208381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Write over the extra field, marking foo.gz as complete. 209381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 210381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Recovery procedure: 211381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - If not a replace recovery, read in the foo.add file, and provide that data 212381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes to the appropriate recovery below. If there is no foo.add file, provide 213381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes a zero data length to the recovery. In that case, the append recovery 214381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes restores the foo.gz to the previous compressed + uncompressed data state. 215381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes For the the compress recovery, a missing foo.add file results in foo.gz 216381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes being restored to the previous compressed-only data state. 217381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Append recovery: 218381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Pick up append at + step above 219381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Compress recovery: 220381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Pick up compress at * step above 221381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Replace recovery: 222381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Pick up compress at @ step above 223381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes - Log the repair with a date stamp in foo.repairs 224381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes */ 225381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 226381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <sys/types.h> 227381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <stdio.h> /* rename, fopen, fprintf, fclose */ 228381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <stdlib.h> /* malloc, free */ 229381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <string.h> /* strlen, strrchr, strcpy, strncpy, strcmp */ 230381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <fcntl.h> /* open */ 231381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <unistd.h> /* lseek, read, write, close, unlink, sleep, */ 232381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* ftruncate, fsync */ 233381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <errno.h> /* errno */ 234381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <time.h> /* time, ctime */ 235381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <sys/stat.h> /* stat */ 236381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include <sys/time.h> /* utimes */ 237381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include "zlib.h" /* crc32 */ 238381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 239381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#include "gzlog.h" /* header for external access */ 2409e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 2419e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project#define local static 242381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughestypedef unsigned int uint; 243381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughestypedef unsigned long ulong; 244381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 245381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Macro for debugging to deterministically force recovery operations */ 246381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#ifdef DEBUG 247381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes #include <setjmp.h> /* longjmp */ 248381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes jmp_buf gzlog_jump; /* where to go back to */ 249381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int gzlog_bail = 0; /* which point to bail at (1..8) */ 250381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int gzlog_count = -1; /* number of times through to wait */ 251381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes# define BAIL(n) do { if (n == gzlog_bail && gzlog_count-- == 0) \ 252381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes longjmp(gzlog_jump, gzlog_bail); } while (0) 253381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#else 254381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes# define BAIL(n) 255381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#endif 256381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 257381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* how old the lock file can be in seconds before considering it stale */ 258381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define PATIENCE 300 259381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 260381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* maximum stored block size in Kbytes -- must be in 1..63 */ 261381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define MAX_STORE 16 2629e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 263381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* number of stored Kbytes to trigger compression (must be >= 32 to allow 264381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes dictionary construction, and <= 204 * MAX_STORE, in order for >> 10 to 265381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes discard the stored block headers contribution of five bytes each) */ 266381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define TRIGGER 1024 267381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 268381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* size of a deflate dictionary (this cannot be changed) */ 269381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define DICT 32768U 270381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 271381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* values for the operation (2 bits) */ 272381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define NO_OP 0 273381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define APPEND_OP 1 274381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define COMPRESS_OP 2 275381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define REPLACE_OP 3 276381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 277381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* macros to extract little-endian integers from an unsigned byte buffer */ 278381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define PULL2(p) ((p)[0]+((uint)((p)[1])<<8)) 279381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define PULL4(p) (PULL2(p)+((ulong)PULL2(p+2)<<16)) 280381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define PULL8(p) (PULL4(p)+((off_t)PULL4(p+4)<<32)) 281381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 282381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* macros to store integers into a byte buffer in little-endian order */ 283381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define PUT2(p,a) do {(p)[0]=a;(p)[1]=(a)>>8;} while(0) 284381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define PUT4(p,a) do {PUT2(p,a);PUT2(p+2,a>>16);} while(0) 285381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define PUT8(p,a) do {PUT4(p,a);PUT4(p+4,a>>32);} while(0) 286381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 287381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* internal structure for log information */ 288381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define LOGID "\106\035\172" /* should be three non-zero characters */ 289381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughesstruct log { 290381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes char id[4]; /* contains LOGID to detect inadvertent overwrites */ 291381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int fd; /* file descriptor for .gz file, opened read/write */ 292381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes char *path; /* allocated path, e.g. "/var/log/foo" or "foo" */ 293381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes char *end; /* end of path, for appending suffices such as ".gz" */ 294381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes off_t first; /* offset of first stored block first length byte */ 295381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int back; /* location of first block id in bits back from first */ 296381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes uint stored; /* bytes currently in last stored block */ 297381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes off_t last; /* offset of last stored block first length byte */ 298381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ulong ccrc; /* crc of compressed data */ 299381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ulong clen; /* length (modulo 2^32) of compressed data */ 300381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ulong tcrc; /* crc of total data */ 301381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ulong tlen; /* length (modulo 2^32) of total data */ 302381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes time_t lock; /* last modify time of our lock file */ 303381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes}; 304381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 305381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* gzip header for gzlog */ 306381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal unsigned char log_gzhead[] = { 307381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0x1f, 0x8b, /* magic gzip id */ 308381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 8, /* compression method is deflate */ 309381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 4, /* there is an extra field (no file name) */ 310381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0, 0, 0, 0, /* no modification time provided */ 311381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0, 0xff, /* no extra flags, no OS specified */ 312381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 39, 0, 'a', 'p', 35, 0 /* extra field with "ap" subfield */ 313381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* 35 is EXTRA, 39 is EXTRA + 4 */ 314381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes}; 315381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 316381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define HEAD sizeof(log_gzhead) /* should be 16 */ 317381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 318381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* initial gzip extra field content (52 == HEAD + EXTRA + 1) */ 319381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal unsigned char log_gzext[] = { 320381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 52, 0, 0, 0, 0, 0, 0, 0, /* offset of first stored block length */ 321381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 52, 0, 0, 0, 0, 0, 0, 0, /* offset of last stored block length */ 322381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0, 0, 0, 0, 0, 0, 0, 0, /* compressed data crc and length */ 323381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0, 0, 0, 0, 0, 0, 0, 0, /* total data crc and length */ 324381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0, 0, /* final stored block data length */ 325381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 5 /* op is NO_OP, last bit 8 bits back */ 326381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes}; 327381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 328381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define EXTRA sizeof(log_gzext) /* should be 35 */ 329381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 330381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* initial gzip data and trailer */ 331381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal unsigned char log_gzbody[] = { 332381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 1, 0, 0, 0xff, 0xff, /* empty stored block (last) */ 333381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0, 0, 0, 0, /* crc */ 334381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0, 0, 0, 0 /* uncompressed length */ 335381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes}; 336381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 337381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes#define BODY sizeof(log_gzbody) 338381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 339381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Exclusively create foo.lock in order to negotiate exclusive access to the 340381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.* files. If the modify time of an existing lock file is greater than 341381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PATIENCE seconds in the past, then consider the lock file to have been 342381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes abandoned, delete it, and try the exclusive create again. Save the lock 343381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes file modify time for verification of ownership. Return 0 on success, or -1 344381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes on failure, usually due to an access restriction or invalid path. Note that 345381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if stat() or unlink() fails, it may be due to another process noticing the 346381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes abandoned lock file a smidge sooner and deleting it, so those are not 347381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes flagged as an error. */ 348381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_lock(struct log *log) 3499e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 350381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int fd; 351381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes struct stat st; 3529e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 353381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".lock"); 354381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes while ((fd = open(log->path, O_CREAT | O_EXCL, 0644)) < 0) { 355381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (errno != EEXIST) 356381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 357381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (stat(log->path, &st) == 0 && time(NULL) - st.st_mtime > PATIENCE) { 358381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unlink(log->path); 359381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes continue; 360381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 361381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes sleep(2); /* relinquish the CPU for two seconds while waiting */ 362381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 363381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes close(fd); 364381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (stat(log->path, &st) == 0) 365381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->lock = st.st_mtime; 366381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return 0; 367381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes} 3689e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 369381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Update the modify time of the lock file to now, in order to prevent another 370381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes task from thinking that the lock is stale. Save the lock file modify time 371381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes for verification of ownership. */ 372381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal void log_touch(struct log *log) 373381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes{ 374381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes struct stat st; 375381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 376381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".lock"); 377381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes utimes(log->path, NULL); 378381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (stat(log->path, &st) == 0) 379381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->lock = st.st_mtime; 3809e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project} 3819e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 382381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Check the log file modify time against what is expected. Return true if 383381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes this is not our lock. If it is our lock, touch it to keep it. */ 384381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_check(struct log *log) 3859e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 386381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes struct stat st; 387381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 388381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".lock"); 389381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (stat(log->path, &st) || st.st_mtime != log->lock) 390381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return 1; 391381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_touch(log); 392381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return 0; 3939e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project} 3949e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 395381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Unlock a previously acquired lock, but only if it's ours. */ 396381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal void log_unlock(struct log *log) 3979e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 398381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_check(log)) 399381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return; 400381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".lock"); 401381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unlink(log->path); 402381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->lock = 0; 4039e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project} 4049e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 405381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Check the gzip header and read in the extra field, filling in the values in 406381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the log structure. Return op on success or -1 if the gzip header was not as 407381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes expected. op is the current operation in progress last written to the extra 408381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes field. This assumes that the gzip file has already been opened, with the 409381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes file descriptor log->fd. */ 410381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_head(struct log *log) 4119e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 412381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int op; 413381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unsigned char buf[HEAD + EXTRA]; 4149e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 415381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (lseek(log->fd, 0, SEEK_SET) < 0 || 416381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes read(log->fd, buf, HEAD + EXTRA) != HEAD + EXTRA || 417381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes memcmp(buf, log_gzhead, HEAD)) { 418381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 419381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 420381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->first = PULL8(buf + HEAD); 421381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->last = PULL8(buf + HEAD + 8); 422381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->ccrc = PULL4(buf + HEAD + 16); 423381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->clen = PULL4(buf + HEAD + 20); 424381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->tcrc = PULL4(buf + HEAD + 24); 425381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->tlen = PULL4(buf + HEAD + 28); 426381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->stored = PULL2(buf + HEAD + 32); 427381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->back = 3 + (buf[HEAD + 34] & 7); 428381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes op = (buf[HEAD + 34] >> 3) & 3; 429381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return op; 4309e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project} 4319e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 432381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Write over the extra field contents, marking the operation as op. Use fsync 433381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes to assure that the device is written to, and in the requested order. This 434381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes operation, and only this operation, is assumed to be atomic in order to 435381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes assure that the log is recoverable in the event of an interruption at any 436381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes point in the process. Return -1 if the write to foo.gz failed. */ 437381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_mark(struct log *log, int op) 4389e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 439381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int ret; 440381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unsigned char ext[EXTRA]; 4419e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 442381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT8(ext, log->first); 443381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT8(ext + 8, log->last); 444381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT4(ext + 16, log->ccrc); 445381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT4(ext + 20, log->clen); 446381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT4(ext + 24, log->tcrc); 447381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT4(ext + 28, log->tlen); 448381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT2(ext + 32, log->stored); 449381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ext[34] = log->back - 3 + (op << 3); 450381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes fsync(log->fd); 451381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ret = lseek(log->fd, HEAD, SEEK_SET) < 0 || 452381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes write(log->fd, ext, EXTRA) != EXTRA ? -1 : 0; 453381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes fsync(log->fd); 454381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return ret; 4559e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project} 4569e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 457381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Rewrite the last block header bits and subsequent zero bits to get to a byte 458381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes boundary, setting the last block bit if last is true, and then write the 459381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes remainder of the stored block header (length and one's complement). Leave 460381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the file pointer after the end of the last stored block data. Return -1 if 461381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes there is a read or write failure on the foo.gz file */ 462381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_last(struct log *log, int last) 4639e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 464381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int back, len, mask; 465381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unsigned char buf[6]; 466381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 467381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* determine the locations of the bytes and bits to modify */ 468381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes back = log->last == log->first ? log->back : 8; 469381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes len = back > 8 ? 2 : 1; /* bytes back from log->last */ 470381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes mask = 0x80 >> ((back - 1) & 7); /* mask for block last-bit */ 471381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 472381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* get the byte to modify (one or two back) into buf[0] -- don't need to 473381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes read the byte if the last-bit is eight bits back, since in that case 474381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the entire byte will be modified */ 475381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes buf[0] = 0; 476381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (back != 8 && (lseek(log->fd, log->last - len, SEEK_SET) < 0 || 477381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes read(log->fd, buf, 1) != 1)) 478381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 479381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 480381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* change the last-bit of the last stored block as requested -- note 481381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes that all bits above the last-bit are set to zero, per the type bits 482381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes of a stored block being 00 and per the convention that the bits to 483381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes bring the stream to a byte boundary are also zeros */ 484381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes buf[1] = 0; 485381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes buf[2 - len] = (*buf & (mask - 1)) + (last ? mask : 0); 4869e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 487381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* write the modified stored block header and lengths, move the file 488381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes pointer to after the last stored block data */ 489381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT2(buf + 2, log->stored); 490381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT2(buf + 4, log->stored ^ 0xffff); 491381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return lseek(log->fd, log->last - len, SEEK_SET) < 0 || 492381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes write(log->fd, buf + 2 - len, len + 4) != len + 4 || 493381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes lseek(log->fd, log->stored, SEEK_CUR) < 0 ? -1 : 0; 494381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes} 495381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 496381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Append len bytes from data to the locked and open log file. len may be zero 497381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if recovering and no .add file was found. In that case, the previous state 498381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes of the foo.gz file is restored. The data is appended uncompressed in 499381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes deflate stored blocks. Return -1 if there was an error reading or writing 500381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the foo.gz file. */ 501381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_append(struct log *log, unsigned char *data, size_t len) 502381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes{ 503381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes uint put; 504381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes off_t end; 505381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unsigned char buf[8]; 506381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 507381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* set the last block last-bit and length, in case recovering an 508381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes interrupted append, then position the file pointer to append to the 509381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes block */ 510381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_last(log, 1)) 511381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 512381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 513381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* append, adding stored blocks and updating the offset of the last stored 514381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes block as needed, and update the total crc and length */ 515381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes while (len) { 516381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* append as much as we can to the last block */ 517381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes put = (MAX_STORE << 10) - log->stored; 518381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (put > len) 519381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes put = (uint)len; 520381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (put) { 521381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (write(log->fd, data, put) != put) 522381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 523381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes BAIL(1); 524381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->tcrc = crc32(log->tcrc, data, put); 525381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->tlen += put; 526381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->stored += put; 527381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes data += put; 528381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes len -= put; 529381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 530381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 531381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* if we need to, add a new empty stored block */ 532381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (len) { 533381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* mark current block as not last */ 534381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_last(log, 0)) 535381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 536381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 537381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* point to new, empty stored block */ 538381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->last += 4 + log->stored + 1; 539381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->stored = 0; 540381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 541381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 542381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* mark last block as last, update its length */ 543381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_last(log, 1)) 544381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 545381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes BAIL(2); 5469e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 547381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 548381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* write the new crc and length trailer, and truncate just in case (could 549381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes be recovering from partial append with a missing foo.add file) */ 550381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT4(buf, log->tcrc); 551381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT4(buf + 4, log->tlen); 552381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (write(log->fd, buf, 8) != 8 || 553381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes (end = lseek(log->fd, 0, SEEK_CUR)) < 0 || ftruncate(log->fd, end)) 554381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 555381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 556381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* write the extra field, marking the log file as done, delete .add file */ 557381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_mark(log, NO_OP)) 558381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 559381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".add"); 560381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unlink(log->path); /* ignore error, since may not exist */ 561381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return 0; 5629e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project} 5639e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 564381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Replace the foo.dict file with the foo.temp file. Also delete the foo.add 565381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes file, since the compress operation may have been interrupted before that was 566381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes done. Returns 1 if memory could not be allocated, or -1 if reading or 567381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes writing foo.gz fails, or if the rename fails for some reason other than 568381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.temp not existing. foo.temp not existing is a permitted error, since 569381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes the replace operation may have been interrupted after the rename is done, 570381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes but before foo.gz is marked as complete. */ 571381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_replace(struct log *log) 5729e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 573381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int ret; 574381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes char *dest; 575381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 576381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* delete foo.add file */ 577381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".add"); 578381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unlink(log->path); /* ignore error, since may not exist */ 579381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes BAIL(3); 580381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 581381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* rename foo.name to foo.dict, replacing foo.dict if it exists */ 582381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".dict"); 583381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes dest = malloc(strlen(log->path) + 1); 584381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (dest == NULL) 585381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -2; 586381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(dest, log->path); 587381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".temp"); 588381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ret = rename(log->path, dest); 589381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes free(dest); 590381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (ret && errno != ENOENT) 591381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 592381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes BAIL(4); 5939e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 594381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* mark the foo.gz file as done */ 595381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return log_mark(log, NO_OP); 596381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes} 597381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 598381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Compress the len bytes at data and append the compressed data to the 599381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes foo.gz deflate data immediately after the previous compressed data. This 600381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes overwrites the previous uncompressed data, which was stored in foo.add 601381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes and is the data provided in data[0..len-1]. If this operation is 602381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes interrupted, it picks up at the start of this routine, with the foo.add 603381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes file read in again. If there is no data to compress (len == 0), then we 604381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes simply terminate the foo.gz file after the previously compressed data, 605381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes appending a final empty stored block and the gzip trailer. Return -1 if 606381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes reading or writing the log.gz file failed, or -2 if there was a memory 607381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes allocation failure. */ 608381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_compress(struct log *log, unsigned char *data, size_t len) 609381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes{ 610381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int fd; 611381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes uint got, max; 612381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ssize_t dict; 613381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes off_t end; 614381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes z_stream strm; 615381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unsigned char buf[DICT]; 616381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 617381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* compress and append compressed data */ 618381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (len) { 619381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* set up for deflate, allocating memory */ 620381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strm.zalloc = Z_NULL; 621381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strm.zfree = Z_NULL; 622381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strm.opaque = Z_NULL; 623381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (deflateInit2(&strm, Z_DEFAULT_COMPRESSION, Z_DEFLATED, -15, 8, 624381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Z_DEFAULT_STRATEGY) != Z_OK) 625381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -2; 626381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 627381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* read in dictionary (last 32K of data that was compressed) */ 628381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".dict"); 629381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes fd = open(log->path, O_RDONLY, 0); 630381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (fd >= 0) { 631381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes dict = read(fd, buf, DICT); 632381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes close(fd); 633381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (dict < 0) { 634381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes deflateEnd(&strm); 635381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 636381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 637381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (dict) 638381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes deflateSetDictionary(&strm, buf, (uint)dict); 639381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 640381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_touch(log); 641381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 642381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* prime deflate with last bits of previous block, position write 643381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes pointer to write those bits and overwrite what follows */ 644381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (lseek(log->fd, log->first - (log->back > 8 ? 2 : 1), 645381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes SEEK_SET) < 0 || 646381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes read(log->fd, buf, 1) != 1 || lseek(log->fd, -1, SEEK_CUR) < 0) { 647381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes deflateEnd(&strm); 648381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 649381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 650381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes deflatePrime(&strm, (8 - log->back) & 7, *buf); 651381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 652381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* compress, finishing with a partial non-last empty static block */ 653381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strm.next_in = data; 654381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes max = (((uint)0 - 1) >> 1) + 1; /* in case int smaller than size_t */ 655381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes do { 656381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strm.avail_in = len > max ? max : (uint)len; 657381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes len -= strm.avail_in; 658381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes do { 659381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strm.avail_out = DICT; 660381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strm.next_out = buf; 661381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes deflate(&strm, len ? Z_NO_FLUSH : Z_PARTIAL_FLUSH); 662381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes got = DICT - strm.avail_out; 663381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (got && write(log->fd, buf, got) != got) { 664381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes deflateEnd(&strm); 665381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 666381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 667381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_touch(log); 668381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } while (strm.avail_out == 0); 669381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } while (len); 670381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes deflateEnd(&strm); 671381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes BAIL(5); 672381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 673381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* find start of empty static block -- scanning backwards the first one 674381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes bit is the second bit of the block, if the last byte is zero, then 675381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes we know the byte before that has a one in the top bit, since an 676381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes empty static block is ten bits long */ 677381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if ((log->first = lseek(log->fd, -1, SEEK_CUR)) < 0 || 678381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes read(log->fd, buf, 1) != 1) 679381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 680381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->first++; 681381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (*buf) { 682381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->back = 1; 683381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes while ((*buf & ((uint)1 << (8 - log->back++))) == 0) 684381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ; /* guaranteed to terminate, since *buf != 0 */ 685381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 686381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes else 687381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->back = 10; 688381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 689381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* update compressed crc and length */ 690381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->ccrc = log->tcrc; 691381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->clen = log->tlen; 692381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 693381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes else { 694381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* no data to compress -- fix up existing gzip stream */ 695381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->tcrc = log->ccrc; 696381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->tlen = log->clen; 6979e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 698381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 699381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* complete and truncate gzip stream */ 700381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->last = log->first; 701381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->stored = 0; 702381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT4(buf, log->tcrc); 703381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes PUT4(buf + 4, log->tlen); 704381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_last(log, 1) || write(log->fd, buf, 8) != 8 || 705381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes (end = lseek(log->fd, 0, SEEK_CUR)) < 0 || ftruncate(log->fd, end)) 706381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 707381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes BAIL(6); 708381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 709381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* mark as being in the replace operation */ 710381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_mark(log, REPLACE_OP)) 711381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 712381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 713381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* execute the replace operation and mark the file as done */ 714381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return log_replace(log); 7159e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project} 7169e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 717381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* log a repair record to the .repairs file */ 718381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal void log_log(struct log *log, int op, char *record) 719381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes{ 720381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes time_t now; 721381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes FILE *rec; 7229e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 723381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes now = time(NULL); 724381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".repairs"); 725381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes rec = fopen(log->path, "a"); 726381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (rec == NULL) 727381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return; 728381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes fprintf(rec, "%.24s %s recovery: %s\n", ctime(&now), op == APPEND_OP ? 729381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes "append" : (op == COMPRESS_OP ? "compress" : "replace"), record); 730381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes fclose(rec); 731381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return; 732381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes} 733381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 734381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Recover the interrupted operation op. First read foo.add for recovering an 735381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes append or compress operation. Return -1 if there was an error reading or 736381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes writing foo.gz or reading an existing foo.add, or -2 if there was a memory 737381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes allocation failure. */ 738381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_recover(struct log *log, int op) 7399e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 740381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int fd, ret = 0; 741381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unsigned char *data = NULL; 742381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes size_t len = 0; 743381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes struct stat st; 7449e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 745381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* log recovery */ 746381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_log(log, op, "start"); 747381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 748381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* load foo.add file if expected and present */ 749381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (op == APPEND_OP || op == COMPRESS_OP) { 750381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".add"); 751381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (stat(log->path, &st) == 0 && st.st_size) { 752381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes len = (size_t)(st.st_size); 75304351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes if ((off_t)len != st.st_size || 75404351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes (data = malloc(st.st_size)) == NULL) { 755381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_log(log, op, "allocation failure"); 756381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -2; 757381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 758381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if ((fd = open(log->path, O_RDONLY, 0)) < 0) { 759381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_log(log, op, ".add file read failure"); 760381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 761381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 76204351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes ret = (size_t)read(fd, data, len) != len; 763381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes close(fd); 764381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (ret) { 765381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_log(log, op, ".add file read failure"); 766381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 767381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 768381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_log(log, op, "loaded .add file"); 769381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 770381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes else 771381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_log(log, op, "missing .add file!"); 772381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } 773381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 774381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* recover the interrupted operation */ 775381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes switch (op) { 776381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes case APPEND_OP: 777381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ret = log_append(log, data, len); 778381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes break; 779381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes case COMPRESS_OP: 780381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ret = log_compress(log, data, len); 781381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes break; 782381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes case REPLACE_OP: 783381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ret = log_replace(log); 7849e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 785381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 786381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* log status */ 787381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_log(log, op, ret ? "failure" : "complete"); 788381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 789381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* clean up */ 790381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (data != NULL) 791381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes free(data); 792381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return ret; 793381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes} 794381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 795381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Close the foo.gz file (if open) and release the lock. */ 796381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal void log_close(struct log *log) 797381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes{ 798381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log->fd >= 0) 7999e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project close(log->fd); 800381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->fd = -1; 801381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_unlock(log); 802381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes} 803381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 804381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* Open foo.gz, verify the header, and load the extra field contents, after 805381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes first creating the foo.lock file to gain exclusive access to the foo.* 806381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes files. If foo.gz does not exist or is empty, then write the initial header, 807381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes extra, and body content of an empty foo.gz log file. If there is an error 808381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes creating the lock file due to access restrictions, or an error reading or 809381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes writing the foo.gz file, or if the foo.gz file is not a proper log file for 810381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes this object (e.g. not a gzip file or does not contain the expected extra 811381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes field), then return true. If there is an error, the lock is released. 812381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes Otherwise, the lock is left in place. */ 813381716e9396b55b1adb8235b020c37344f60ab07Elliott Hugheslocal int log_open(struct log *log) 814381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes{ 815381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int op; 816381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 817381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* release open file resource if left over -- can occur if lock lost 818381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes between gzlog_open() and gzlog_write() */ 819381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log->fd >= 0) 820381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes close(log->fd); 821381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->fd = -1; 822381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 823381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* negotiate exclusive access */ 824381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_lock(log) < 0) 825381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 826381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 827381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* open the log file, foo.gz */ 828381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".gz"); 829381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->fd = open(log->path, O_RDWR | O_CREAT, 0644); 830381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log->fd < 0) { 831381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_close(log); 832381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 8339e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 8349e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 835381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* if new, initialize foo.gz with an empty log, delete old dictionary */ 8369e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project if (lseek(log->fd, 0, SEEK_END) == 0) { 837381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (write(log->fd, log_gzhead, HEAD) != HEAD || 838381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes write(log->fd, log_gzext, EXTRA) != EXTRA || 839381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes write(log->fd, log_gzbody, BODY) != BODY) { 840381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_close(log); 841381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 8429e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 843381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".dict"); 844381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unlink(log->path); 8459e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 8469e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 847381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* verify log file and load extra field information */ 848381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if ((op = log_head(log)) < 0) { 849381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_close(log); 850381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 8519e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 8529e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 853381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* check for interrupted process and if so, recover */ 854381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (op != NO_OP && log_recover(log, op)) { 855381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_close(log); 856381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 8579e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 858381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 859381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* touch the lock file to prevent another process from grabbing it */ 860381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_touch(log); 861381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return 0; 862381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes} 863381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 864381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* See gzlog.h for the description of the external methods below */ 865381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughesgzlog *gzlog_open(char *path) 866381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes{ 867381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes size_t n; 868381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes struct log *log; 869381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 870381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* check arguments */ 871381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (path == NULL || *path == 0) 8729e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project return NULL; 8739e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 874381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* allocate and initialize log structure */ 875381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log = malloc(sizeof(struct log)); 876381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log == NULL) 877381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return NULL; 878381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->id, LOGID); 879381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->fd = -1; 880381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 881381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* save path and end of path for name construction */ 882381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes n = strlen(path); 883381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->path = malloc(n + 9); /* allow for ".repairs" */ 884381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log->path == NULL) { 885381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes free(log); 8869e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project return NULL; 8879e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 888381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->path, path); 889381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->end = log->path + n; 890381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 891381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* gain exclusive access and verify log file -- may perform a 892381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes recovery operation if needed */ 893381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_open(log)) { 894381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes free(log->path); 895381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes free(log); 8969e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project return NULL; 8979e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 8989e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 899381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* return pointer to log structure */ 900381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return log; 9019e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project} 9029e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 903381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* gzlog_compress() return values: 904381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0: all good 905381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes -1: file i/o error (usually access issue) 906381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes -2: memory allocation failure 907381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes -3: invalid log pointer argument */ 908381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughesint gzlog_compress(gzlog *logd) 9099e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 910381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int fd, ret; 911381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes uint block; 912381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes size_t len, next; 913381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes unsigned char *data, buf[5]; 914381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes struct log *log = logd; 9159e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 916381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* check arguments */ 91704351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes if (log == NULL || strcmp(log->id, LOGID)) 918381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -3; 9199e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 920381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* see if we lost the lock -- if so get it again and reload the extra 921381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes field information (it probably changed), recover last operation if 922381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes necessary */ 923381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_check(log) && log_open(log)) 924381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 9259e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 926381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* create space for uncompressed data */ 927381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes len = ((size_t)(log->last - log->first) & ~(((size_t)1 << 10) - 1)) + 928381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->stored; 929381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if ((data = malloc(len)) == NULL) 930381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -2; 9319e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 932381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* do statement here is just a cheap trick for error handling */ 933381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes do { 934381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* read in the uncompressed data */ 935381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (lseek(log->fd, log->first - 1, SEEK_SET) < 0) 9369e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project break; 937381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes next = 0; 938381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes while (next < len) { 939381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (read(log->fd, buf, 5) != 5) 9409e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project break; 941381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes block = PULL2(buf + 1); 942381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (next + block > len || 943381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes read(log->fd, (char *)data + next, block) != block) 9449e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project break; 945381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes next += block; 9469e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project } 947381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (lseek(log->fd, 0, SEEK_CUR) != log->last + 4 + log->stored) 948381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes break; 949381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_touch(log); 9509e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 951381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* write the uncompressed data to the .add file */ 952381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".add"); 953381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes fd = open(log->path, O_WRONLY | O_CREAT | O_TRUNC, 0644); 954381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (fd < 0) 955381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes break; 95604351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes ret = (size_t)write(fd, data, len) != len; 957381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (ret | close(fd)) 958381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes break; 959381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_touch(log); 9609e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 961381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* write the dictionary for the next compress to the .temp file */ 962381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".temp"); 963381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes fd = open(log->path, O_WRONLY | O_CREAT | O_TRUNC, 0644); 964381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (fd < 0) 965381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes break; 966381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes next = DICT > len ? len : DICT; 96704351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes ret = (size_t)write(fd, (char *)data + len - next, next) != next; 968381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (ret | close(fd)) 969381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes break; 970381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_touch(log); 9719e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 972381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* roll back to compressed data, mark the compress in progress */ 973381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->last = log->first; 974381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log->stored = 0; 975381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_mark(log, COMPRESS_OP)) 976381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes break; 977381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes BAIL(7); 978381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 979381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* compress and append the data (clears mark) */ 980381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes ret = log_compress(log, data, len); 981381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes free(data); 982381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return ret; 983381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes } while (0); 9849e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 985381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* broke out of do above on i/o error */ 986381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes free(data); 987381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 988381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes} 9899e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 990381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* gzlog_write() return values: 991381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0: all good 992381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes -1: file i/o error (usually access issue) 993381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes -2: memory allocation failure 994381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes -3: invalid log pointer argument */ 995381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughesint gzlog_write(gzlog *logd, void *data, size_t len) 9969e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project{ 997381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes int fd, ret; 998381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes struct log *log = logd; 9999e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 1000381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* check arguments */ 100104351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes if (log == NULL || strcmp(log->id, LOGID)) 1002381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -3; 100304351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes if (data == NULL || len <= 0) 1004381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return 0; 10059e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 1006381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* see if we lost the lock -- if so get it again and reload the extra 1007381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes field information (it probably changed), recover last operation if 1008381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes necessary */ 1009381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_check(log) && log_open(log)) 1010381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 10119e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 1012381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* create and write .add file */ 1013381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->end, ".add"); 1014381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes fd = open(log->path, O_WRONLY | O_CREAT | O_TRUNC, 0644); 1015381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (fd < 0) 1016381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 101704351a92ecc8429c999acbfc5dfe5aa8bee1d19dElliott Hughes ret = (size_t)write(fd, data, len) != len; 1018381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (ret | close(fd)) 1019381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 1020381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_touch(log); 10219e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 1022381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* mark log file with append in progress */ 1023381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_mark(log, APPEND_OP)) 1024381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 1025381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes BAIL(8); 10269e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 1027381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* append data (clears mark) */ 1028381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log_append(log, data, len)) 1029381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -1; 10309e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 1031381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* check to see if it's time to compress -- if not, then done */ 1032381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (((log->last - log->first) >> 10) + (log->stored >> 10) < TRIGGER) 1033381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return 0; 1034381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 1035381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* time to compress */ 1036381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return gzlog_compress(log); 1037381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes} 1038381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 1039381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes/* gzlog_close() return values: 1040381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 0: ok 1041381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes -3: invalid log pointer argument */ 1042381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughesint gzlog_close(gzlog *logd) 1043381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes{ 1044381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes struct log *log = logd; 10459e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project 1046381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* check arguments */ 1047381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log == NULL || strcmp(log->id, LOGID)) 1048381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes return -3; 1049381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 1050381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* close the log file and release the lock */ 1051381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes log_close(log); 1052381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes 1053381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes /* free structure and return */ 1054381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes if (log->path != NULL) 1055381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes free(log->path); 1056381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes strcpy(log->id, "bad"); 1057381716e9396b55b1adb8235b020c37344f60ab07Elliott Hughes free(log); 10589e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project return 0; 10599e38dfa2f95fce609707a0941f10af9a785288deThe Android Open Source Project} 1060