1The log file contents are a sequence of 32KB blocks. The only 2exception is that the tail of the file may contain a partial block. 3 4Each block consists of a sequence of records: 5 block := record* trailer? 6 record := 7 checksum: uint32 // crc32c of type and data[] ; little-endian 8 length: uint16 // little-endian 9 type: uint8 // One of FULL, FIRST, MIDDLE, LAST 10 data: uint8[length] 11 12A record never starts within the last six bytes of a block (since it 13won't fit). Any leftover bytes here form the trailer, which must 14consist entirely of zero bytes and must be skipped by readers. 15 16Aside: if exactly seven bytes are left in the current block, and a new 17non-zero length record is added, the writer must emit a FIRST record 18(which contains zero bytes of user data) to fill up the trailing seven 19bytes of the block and then emit all of the user data in subsequent 20blocks. 21 22More types may be added in the future. Some Readers may skip record 23types they do not understand, others may report that some data was 24skipped. 25 26FULL == 1 27FIRST == 2 28MIDDLE == 3 29LAST == 4 30 31The FULL record contains the contents of an entire user record. 32 33FIRST, MIDDLE, LAST are types used for user records that have been 34split into multiple fragments (typically because of block boundaries). 35FIRST is the type of the first fragment of a user record, LAST is the 36type of the last fragment of a user record, and MID is the type of all 37interior fragments of a user record. 38 39Example: consider a sequence of user records: 40 A: length 1000 41 B: length 97270 42 C: length 8000 43A will be stored as a FULL record in the first block. 44 45B will be split into three fragments: first fragment occupies the rest 46of the first block, second fragment occupies the entirety of the 47second block, and the third fragment occupies a prefix of the third 48block. This will leave six bytes free in the third block, which will 49be left empty as the trailer. 50 51C will be stored as a FULL record in the fourth block. 52 53=================== 54 55Some benefits over the recordio format: 56 57(1) We do not need any heuristics for resyncing - just go to next 58block boundary and scan. If there is a corruption, skip to the next 59block. As a side-benefit, we do not get confused when part of the 60contents of one log file are embedded as a record inside another log 61file. 62 63(2) Splitting at approximate boundaries (e.g., for mapreduce) is 64simple: find the next block boundary and skip records until we 65hit a FULL or FIRST record. 66 67(3) We do not need extra buffering for large records. 68 69Some downsides compared to recordio format: 70 71(1) No packing of tiny records. This could be fixed by adding a new 72record type, so it is a shortcoming of the current implementation, 73not necessarily the format. 74 75(2) No compression. Again, this could be fixed by adding new record types. 76