| d7768d5c29e747020531b5f71e7c7cfe980ea861 |
|
01-Mar-2018 |
Amin Hassani <ahassani@google.com> |
Convert majority of size_t to uint64_t for 32-bit compatibility
We need to get puffin to work correctly on 32-bit machines. This patch converts
appropriate size_t types to uint64_t to make sure when we are working with large
files in 32bit machines, nothing breaks.
The 32 bit tests actually found some errors. For example BitReader and BitWriter
should have their size and indices kept in 64 bit integers otherwise a multiply
by 8 (to figure out the bit-based length) will overflow. Although it has not
been tested for puffdiff and puffpatch.
Bug: 73781483
Test: unittests pass
Test: puffhuff on a >5G zlib file passes
Test: build with 32-bit compiler, and puffhuff on >1GB passes
Change-Id: I0234a787e095a7ebf0489966a7b7565b6688496c
/external/puffin/src/main.cc
|
| fb877fc98cc72462d3e58e7d744184908408d72f |
|
21-Feb-2018 |
Amin Hassani <ahassani@google.com> |
Start detecting file types
This patch adds option for detecting file types based on the extension of the
given file paths. However this file type can be overriden by explicitly passing
src_file_type or dst_file_type. Btw, In turns out it is a good idea to add
dst_file_type flag to so we can better test the patching mechanism.
Bug: 73487244
Test: puffin --verbose --operation=puffdiff --src_file_type=zip --src_file=zipme.zip --dst_file=big.txt.1.gz --dst_file_type=gzip --patch_file=patch.bin
Change-Id: Ib860b68acc822a256bd56cdda30dc59273b878c2
/external/puffin/src/main.cc
|
| 75a7f2c89d1af29d31c1008ed888c665fdcec919 |
|
21-Feb-2018 |
Amin Hassani <ahassani@google.com> |
Locate zlib stream from a buffer
Previously we were locating deflates inside zlib streams based on the given
location of zlib streams. That was not consistent with gzip and zip
streams. This patch changes the internal API and implementation to makes it more
consistent.
Bug: 73487244
Test: unittests
Test: 'brillo_update_payload generate'
Change-Id: Iab0172fe102ab83944f8aaf90d68d9f08f36ae40
/external/puffin/src/main.cc
|
| a4c5ba6acd3a7935a17d586d88ebf32a7c0663ce |
|
15-Feb-2018 |
Amin Hassani <ahassani@google.com> |
Add puffhuff operation to puffin executable
puffhuff operation allows puffing a stream and then huffing it to make
sure we can remake the original stream. We can do this with doing puff
once and followed by a huff operation, but that requires passing puff
stream locations to the huff operation.
Bug: 73487244
Test: unittests pass
Test: puffin --operation=puffhuff creates the original file.
Change-Id: I93c792a5fff4226cc771a1989fb874b4c76cf8ab
/external/puffin/src/main.cc
|
| c44b53e63c7092bfe05c3bafcc7ef87c042f3034 |
|
15-Feb-2018 |
Amin Hassani <ahassani@google.com> |
Fixes some styling and clang-format in main.cc
Bug: 73487244
Test: unitests pass
Change-Id: If11daa6feb7467175c7fe614b8ec8ad10d7e7cf8
/external/puffin/src/main.cc
|
| 4a212ed06e78e5f1468baeba000db2185e208f22 |
|
15-Feb-2018 |
Amin Hassani <ahassani@google.com> |
Parse gzip stream
This patch adds function LocateDeflatesInGzip() for parsing gzip
streams. it returns the byte alignment locations of gzip streams. It
supports multiple member gzip streams.
Bug: 73487244
Test: puffin --src_file_type=file.gzip works;
Test: unittests pass
Change-Id: I239c695440bd490627481794dc73c0e967234595
/external/puffin/src/main.cc
|
| bf609da38478f3de214949628aae8ffd381e7209 |
|
14-Feb-2018 |
Amin Hassani <ahassani@google.com> |
Allow parsing compressed files in puffin
This patch adds a new flag src_file_type the puffin executable to allow
parsing an input file without the need to manually give the deflate
locations. We can later add dst_file_type to do puffpatch between
different types of files, but it is not necessary at this point yet.
Bug: 73487244
Test: puffin with src_file_type of zlib, deflate and zip passed.
Change-Id: I1addc8f1567bda652c4f138f0e9f825eb7ece4f8
/external/puffin/src/main.cc
|
| 846d9f7aff802bb4adb6cc11af1682eff3a7c6f0 |
|
10-Nov-2017 |
Amin Hassani <ahassani@google.com> |
puffin: Remove dst_puff_size
We don't need this flag anymore as we can caculate the size of the puff stream.
BUG=none
TEST=unittests pass;
Change-Id: I955fc30e754bafae5135f87aac0a26ff5698863d
Reviewed-on: https://chromium-review.googlesource.com/763870
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Sen Jiang <senj@chromium.org>
/external/puffin/src/main.cc
|
| 58ffb83ad394baf5a757c2c89c2b111cabffa479 |
|
16-Oct-2017 |
Amin Hassani <ahassani@google.com> |
puffin: Cache puff stream in puffpatch
Currently without caching the puff stream, the OS updates can be very slow. For
example, a back of envelope calculation can show that if we don't cache the puff
stream, we may need to read and puff the entire deflate stream (about 500 MB)
160 times which can be very slow. And examination shows that updates can take
more than one hour to finish on a veyron_minnie. It may not be reasonable to
keep memory usage low but keep the CPU usage high for extended periods of
time. The other rationale behind caching the puff stream is that if we only use
bsdiff, then we have to keep large patches in memory when applying the
patch. But since puffin reduced the patch size by a large factor (so to keep a
much smaller patch in memory), then it won't be as bad to cache some memory for
faster patching. It is basically a trade-off between CPU time and memory. We may
end up using the same amount of memory (which in practice is not going to
happen, we will use much less), but the payload patch size is much smaller.
This patch modifies PuffinStream to cache only the Puff Buffers up to a maximum
passed total memory size as a parameter to puffpatch. This acts as an LRU, the
least recently used puff buffers are ejected when the cache is full.
BUG=chromium:775072
TEST=unittests passed; brillo_update_payload verfiy passes; cros flash with delta updates passes;
Change-Id: I2a390eb35eb5449577cb5e2b1e857bcdca093f68
Reviewed-on: https://chromium-review.googlesource.com/722452
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>
/external/puffin/src/main.cc
|
| 37da9ebc385393391f4299fa3aa0872c533b9255 |
|
17-Oct-2017 |
Amin Hassani <ahassani@google.com> |
puffin: Move trivial streams out of puffpatch.
Currently FileStream and MemoryStream are part of puffpatch library but
they are not used in puffpatch. This CL moves them into a new header and
source and just adds them in puffdiff. These streams will be used in
puffdiff and utility functions. This also reduced the puffpatch library
size another 4KB.
BUG=chromium:775072
TEST=unittests passed;
Change-Id: I69d9f52b094d5753bf299e38c57d7ce01a00d44f
Reviewed-on: https://chromium-review.googlesource.com/726944
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Sen Jiang <senj@chromium.org>
/external/puffin/src/main.cc
|
| 27574e9e83eaefe241efe97c482f1f3e05c6f219 |
|
10-Oct-2017 |
Amin Hassani <ahassani@google.com> |
puffin: Add ExtentStream
We need some sort of extent stream for the update payload tests in
update_engine. These scripts are run as a part of canary builds. This CL
adds a new class ExtentStream that is only used in main.cc for applying
patches. This class should not be shipped to the client devices as part
of puffpatch nor it needs to be in puffdiff.
BUG=chromium:768461
TEST=unittest pass;
Change-Id: Ic482be8e9b6b7e7f12f5f796de1d96d57275097b
Reviewed-on: https://chromium-review.googlesource.com/714453
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Sen Jiang <senj@chromium.org>
/external/puffin/src/main.cc
|
| 7074da6508f1ab7cb860342e2434c5f5804bafbf |
|
01-Oct-2017 |
Amin Hassani <ahassani@google.com> |
puffin: Introduce deflate bit addressing
This CL changes the addressing scheme of deflate blocks from Byte to
Bit. This CL allows breaking a given deflate stream into its
subblocks. A proper function is added to located the deflate subblocks
from a deflate stream. Proper unittests is added to check for new
functionality and its corner cases.
BUG=chromium:768505
TEST=unittests pass; brillo_update_payload {generate|verify} pass;
Change-Id: I9e953d0b7cafca417c09ec56a49ea2e93f027304
Reviewed-on: https://chromium-review.googlesource.com/703681
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>
/external/puffin/src/main.cc
|
| 26bcfdd46ee56055b61b6e8f23f0ff57626a95a7 |
|
30-Sep-2017 |
Amin Hassani <ahassani@google.com> |
puffin: Use empty puff writer
There was a piece of code (Puffer.Puff) that was only being used in
puffdiff but it was also being shipped with puffpatch too. That code
figures out the size of the puff by puffing into a buffer, There was two
problem with that. First, the code shouldn't have been mixed with the
client code, and second, it needs retry of the Puffer.PuffDeflate to do
the puffing again if the buffer size was small. This patch fixes that
problem by removing that function and adding checks in the
Puffer.PuffDeflate to not check for boundary or write into the buffer if
the given buffer size was nullptr. This way we can create a PuffWriter
with null buffer that can count the number of bytes needed for a puff
buffer. Along with, all PuffDeflate and HuffDeflate functions based on
the byte array buffer where removed and equivalent functions based on
BitReader/BitWriter/PuffReader/PuffWriter were made public.
Additionally this CL, adds a new function for finding the location of
Puffs using the aforementioned functionality in the utils.cc (which will
only be in used in puffdiff).
Furthermore, this CL fixes a long undiscovered bug, for when the length
of the literals was exactly 127. The PuffWriter would write a wrong
value to the output and this would cause problem. The unittests for this
bug was also added. Since this bug was in PuffWriter.cc and most of the
changes are in PuffWriter.cc this was a good candidate for fixing this
bug.
BUG=chromium:768505
TEST=unittests pass; brillo_update_payload {generate|verify} pass;
Change-Id: Ie5eb12dc637623b8f3ed08709590a323d9f830f8
Reviewed-on: https://chromium-review.googlesource.com/703680
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Sen Jiang <senj@chromium.org>
/external/puffin/src/main.cc
|
| c3e6b533aa435516b788ae353a43602c57936afd |
|
08-Mar-2017 |
Amin Hassani <ahassani@google.com> |
Puffin: Deterministic patching tool for deflate streams.
This is a deterministic semi-recompression based patching tool for
deflate streams. Its main purpose is to provide necessary tools for
updating the squashfs Android container which is gzipped (deflate
base). For more details refer to go/puffin-recompression.
BUG=chromium:717785
TEST=cros_workon_make --board=amd64-generic puffin --test
CQ-DEPEND=CL:451881
Change-Id: I6f644bf078bedb809d14fba821a12ddf9849712a
Reviewed-on: https://chromium-review.googlesource.com/451861
Commit-Ready: Amin Hassani <ahassani@chromium.org>
Tested-by: Amin Hassani <ahassani@chromium.org>
Reviewed-by: Andrew de los Reyes <adlr@chromium.org>
Reviewed-by: Sen Jiang <senj@chromium.org>
/external/puffin/src/main.cc
|