9dbd802c8c96c3a66873bc600bc7d1374a1d08e5 |
|
31-Aug-2017 |
Orion Hodson <oth@google.com> |
dx: Minor code clean-up A few automated code clean-ups: - Insert missing @Override annotations. - Make fields final where possible. - Use StringBuilder rather StringBuffer for local use. - Clean-up imports. Bug: 65231722 Test: dx/tests/run-all-tests Change-Id: I49810f1d43cb53ad32636a66eda2fadc538ba249
|
bd3b381a74023a63b3713749e4be02429467f789 |
|
13-Nov-2014 |
Peter Jensen <jensenp@google.com> |
Support --num-threads with --multi-dex (take 2) With fix for regression introduced in original commit. The current dx implementation supports options --multi-dex, for applications not fitting within the dex format limitations; and --num-threads=N, triggers concurrent processing of multiple input files. However, the implementation has the following limitations: The --num-threads option is disabled when used together with --multi-dex. The --num-threads option implements concurrency at the level of classpath entries, and does nothing when the classes to be translated are specified with a single classpath element (e.g. single jar output from Proguard). The existing --num-threads implementation may produce indeterministic output. The heuristic used by the --multi-dex option to determine when to rotate the dex output file is overly conservative. The primary objective of this change is: Concurrent translation of classes, independently of input specification format. Support --num-threads=N in both mono- and multi-dex mode. Deterministic class output order. Near optimal use of dex file format capacity. This is accomplished by reorganizing the dx workflow in a pipeline of concurrent phases. read-class | parse-class | translate-class | add-to-dex | convert-dex-to-byte[]; output-dex-files-or-jar To manage dex file rotation (i.e. --multi-dex support), the parse-class and add-to-dex phases are synchronized to prevent forwarding classes to the translate-class phase if it could potentially result in breaking the dex format limitations. The heuristic currently used to estimate the number of indices needed for a class is improved, to minimize the amount of serialization imposed by this feedback mechanism, and to improve the use of dex file capacity. The translate-class and convert-dex-to-byte[] phases are further parallelized with configurable (--num-threads=N option) thread pools. This allow translating classes concurrently, while also performing output conversion in parallel. Separate collector threads are used to collect results from the thread pools in deterministic order. Testing was performed on an Ubuntu system, with 6 cores and 12 hardware threads. The taskset command was used to experimentally establish that running with more than 8 hardware threads does not provide any additional benefit. Experiments shows that the argument to --num-threads should not exceed the lesser of the number of available hardware threads, and 5. Setting it to a higher value results in no additional benefit. The gain is generally larger for larger applications, and not significant for small applications with less than a few thousands classes. Experiments with generated classes shows that for large applications gains as high as 50% may be possible. For an existing real-life application with more than 11k classes, and requiring 2 dex files, a speed-up of 37% was achieved (--num-threads=5, 8 hardware threads, 4g Java heap). A speedup of 31% was observed for another application with ~7 classes. For small applications, use of --num-threads=N>1 doesn’t provide significant benefit. Running with --num-threads=1, the modified dx is slightly faster, but no significant gain is observed unless the application requires multiple dex files. The one case where a significant regression may be observed is when using --num-threads=N>1, with a single hardware thread. This is an inappropriate configuration, even with the current implementation. However, because of the limitations of the current implementation, such configurations may exist. For instance, a configuration using both --multi-dex and --num-threads=5 will currently generate a warning about using the two options together. With the new implementation, the options can legitimately be used together, and could result in an ~20% regression running on a single hardware thread. Note: the current dx implementation, without --num-threads option, is already approximately 50% slower with 1 hardware thread, compared to running with 2 or more. With 2 hardware threads the implementations are practically at par (a little better, or a little worse, depending on the application). Testing: Tested with 6 existing applications ranging in size from 1K - 12K classes. Updated and tested with relevant existing unit tests (one test changed to account for better dex rotation heuristic). Added unit test to test deterministic output. Added unit performance test. By default run script merely validates that --multi-dex and --num-threads can be used together (fast). However, the test is configurable to perform performance test, over sets of generated classes. Signed-off-by: Peter Jensen <jensenp@google.com> (cherry picked from commit 845d9d0eed0f6556e11ee7f7204fda9c8dd41154) (cherry picked from commit dd140a22d90495045024334a91770acaad8e065e) Change-Id: I33a8ea0451efc0af7eb1d72e80cb926d6583d569
|
dd140a22d90495045024334a91770acaad8e065e |
|
13-Nov-2014 |
Peter Jensen <jensenp@google.com> |
Support --num-threads with --multi-dex (take 2) With fix for regression introduced in original commit. The current dx implementation supports options --multi-dex, for applications not fitting within the dex format limitations; and --num-threads=N, triggers concurrent processing of multiple input files. However, the implementation has the following limitations: The --num-threads option is disabled when used together with --multi-dex. The --num-threads option implements concurrency at the level of classpath entries, and does nothing when the classes to be translated are specified with a single classpath element (e.g. single jar output from Proguard). The existing --num-threads implementation may produce indeterministic output. The heuristic used by the --multi-dex option to determine when to rotate the dex output file is overly conservative. The primary objective of this change is: Concurrent translation of classes, independently of input specification format. Support --num-threads=N in both mono- and multi-dex mode. Deterministic class output order. Near optimal use of dex file format capacity. This is accomplished by reorganizing the dx workflow in a pipeline of concurrent phases. read-class | parse-class | translate-class | add-to-dex | convert-dex-to-byte[]; output-dex-files-or-jar To manage dex file rotation (i.e. --multi-dex support), the parse-class and add-to-dex phases are synchronized to prevent forwarding classes to the translate-class phase if it could potentially result in breaking the dex format limitations. The heuristic currently used to estimate the number of indices needed for a class is improved, to minimize the amount of serialization imposed by this feedback mechanism, and to improve the use of dex file capacity. The translate-class and convert-dex-to-byte[] phases are further parallelized with configurable (--num-threads=N option) thread pools. This allow translating classes concurrently, while also performing output conversion in parallel. Separate collector threads are used to collect results from the thread pools in deterministic order. Testing was performed on an Ubuntu system, with 6 cores and 12 hardware threads. The taskset command was used to experimentally establish that running with more than 8 hardware threads does not provide any additional benefit. Experiments shows that the argument to --num-threads should not exceed the lesser of the number of available hardware threads, and 5. Setting it to a higher value results in no additional benefit. The gain is generally larger for larger applications, and not significant for small applications with less than a few thousands classes. Experiments with generated classes shows that for large applications gains as high as 50% may be possible. For an existing real-life application with more than 11k classes, and requiring 2 dex files, a speed-up of 37% was achieved (--num-threads=5, 8 hardware threads, 4g Java heap). A speedup of 31% was observed for another application with ~7 classes. For small applications, use of --num-threads=N>1 doesn’t provide significant benefit. Running with --num-threads=1, the modified dx is slightly faster, but no significant gain is observed unless the application requires multiple dex files. The one case where a significant regression may be observed is when using --num-threads=N>1, with a single hardware thread. This is an inappropriate configuration, even with the current implementation. However, because of the limitations of the current implementation, such configurations may exist. For instance, a configuration using both --multi-dex and --num-threads=5 will currently generate a warning about using the two options together. With the new implementation, the options can legitimately be used together, and could result in an ~20% regression running on a single hardware thread. Note: the current dx implementation, without --num-threads option, is already approximately 50% slower with 1 hardware thread, compared to running with 2 or more. With 2 hardware threads the implementations are practically at par (a little better, or a little worse, depending on the application). Testing: Tested with 6 existing applications ranging in size from 1K - 12K classes. Updated and tested with relevant existing unit tests (one test changed to account for better dex rotation heuristic). Added unit test to test deterministic output. Added unit performance test. By default run script merely validates that --multi-dex and --num-threads can be used together (fast). However, the test is configurable to perform performance test, over sets of generated classes. Signed-off-by: Peter Jensen <jensenp@google.com> (cherry picked from commit 845d9d0eed0f6556e11ee7f7204fda9c8dd41154) Change-Id: I721effa31c3b1a8b427d3a18ec554a19c5e9765b
|
c8b036e3fb5e88eb501e953a8a8838b547f2dab4 |
|
09-Feb-2015 |
Benoit Lamarche <benoitlamarche@google.com> |
Revert "Support --num-threads with --multi-dex" This reverts commit 845d9d0eed0f6556e11ee7f7204fda9c8dd41154. Bug: 19313927 Change-Id: Ia6582a3914cc33762aef74da1f5a6a153c8c0ab2
|
845d9d0eed0f6556e11ee7f7204fda9c8dd41154 |
|
13-Nov-2014 |
Peter Jensen <jensenp@google.com> |
Support --num-threads with --multi-dex The current dx implementation supports options --multi-dex, for applications not fitting within the dex format limitations; and --num-threads=N, triggers concurrent processing of multiple input files. However, the implementation has the following limitations: The --num-threads option is disabled when used together with --multi-dex. The --num-threads option implements concurrency at the level of classpath entries, and does nothing when the classes to be translated are specified with a single classpath element (e.g. single jar output from Proguard). The existing --num-threads implementation may produce indeterministic output. The heuristic used by the --multi-dex option to determine when to rotate the dex output file is overly conservative. The primary objective of this change is: Concurrent translation of classes, independently of input specification format. Support --num-threads=N in both mono- and multi-dex mode. Deterministic class output order. Near optimal use of dex file format capacity. This is accomplished by reorganizing the dx workflow in a pipeline of concurrent phases. read-class | parse-class | translate-class | add-to-dex | convert-dex-to-byte[]; output-dex-files-or-jar To manage dex file rotation (i.e. --multi-dex support), the parse-class and add-to-dex phases are synchronized to prevent forwarding classes to the translate-class phase if it could potentially result in breaking the dex format limitations. The heuristic currently used to estimate the number of indices needed for a class is improved, to minimize the amount of serialization imposed by this feedback mechanism, and to improve the use of dex file capacity. The translate-class and convert-dex-to-byte[] phases are further parallelized with configurable (--num-threads=N option) thread pools. This allow translating classes concurrently, while also performing output conversion in parallel. Separate collector threads are used to collect results from the thread pools in deterministic order. Testing was performed on an Ubuntu system, with 6 cores and 12 hardware threads. The taskset command was used to experimentally establish that running with more than 8 hardware threads does not provide any additional benefit. Experiments shows that the argument to --num-threads should not exceed the lesser of the number of available hardware threads, and 5. Setting it to a higher value results in no additional benefit. The gain is generally larger for larger applications, and not significant for small applications with less than a few thousands classes. Experiments with generated classes shows that for large applications gains as high as 50% may be possible. For an existing real-life application with more than 11k classes, and requiring 2 dex files, a speed-up of 37% was achieved (--num-threads=5, 8 hardware threads, 4g Java heap). A speedup of 31% was observed for another application with ~7 classes. For small applications, use of --num-threads=N>1 doesn’t provide significant benefit. Running with --num-threads=1, the modified dx is slightly faster, but no significant gain is observed unless the application requires multiple dex files. The one case where a significant regression may be observed is when using --num-threads=N>1, with a single hardware thread. This is an inappropriate configuration, even with the current implementation. However, because of the limitations of the current implementation, such configurations may exist. For instance, a configuration using both --multi-dex and --num-threads=5 will currently generate a warning about using the two options together. With the new implementation, the options can legitimately be used together, and could result in an ~20% regression running on a single hardware thread. Note: the current dx implementation, without --num-threads option, is already approximately 50% slower with 1 hardware thread, compared to running with 2 or more. With 2 hardware threads the implementations are practically at par (a little better, or a little worse, depending on the application). Testing: Tested with 6 existing applications ranging in size from 1K - 12K classes. Updated and tested with relevant existing unit tests (one test changed to account for better dex rotation heuristic). Added unit test to test deterministic output. Added unit performance test. By default run script merely validates that --multi-dex and --num-threads can be used together (fast). However, the test is configurable to perform performance test, over sets of generated classes. Change-Id: Ic2d11c422396e97171c2e6ceae9477113e261b8e Signed-off-by: Peter Jensen <jensenp@google.com>
|
fe107fb6e3f308ac5174ebdc5a794ee880c741d9 |
|
17-Sep-2011 |
Jesse Wilson <jessewilson@google.com> |
Mechanical refactoring of dx into two parts. This splits off a new package, com.android.dex that contains code for parsing and modelling dex files. This code is usable both at build time (for compilation and analysis) and at runtime (for introspection). The original package, com.android.dx is the tool that compiles .class files into .dex files. That package also includes utilities for merging and querying dex files, all visible to the dx command line application. In a follow up change I'll move the new com.android.dex package into the libcore/ project, and configure build rules so that it's included in both dx.jar and core.jar. The core.jar will then be able to use it for annotation processing. Change-Id: I3c58cf87e728e4dda1925a0992c1ee7b8130e81a (cherry picked from commit bab4abb07335d162ecdb8091fc395cf84803a580) Conflicts: dx/junit-tests/com/android/dx/util/BitIntSetTest.java dx/junit-tests/com/android/dx/util/BitsTest.java dx/junit-tests/com/android/dx/util/IntListTest.java dx/junit-tests/com/android/dx/util/ListIntSetTest.java dx/src/com/android/dex/Dex.java dx/src/com/android/dex/TableOfContents.java dx/src/com/android/dx/annotations/AnnotationAccess.java dx/src/com/android/dx/command/Main.java dx/src/com/android/dx/dex/code/OutputFinisher.java dx/src/com/android/dx/dex/code/form/Form32s.java dx/src/com/android/dx/dex/code/form/Form33x.java dx/src/com/android/dx/dex/code/form/Form41c.java dx/src/com/android/dx/dex/code/form/Form52c.java dx/src/com/android/dx/gen/DexGenerator.java dx/src/com/android/dx/merge/DexMerger.java
|
de75089fb7216d19e9c22cce4dc62a49513477d3 |
|
09-Jun-2010 |
Carl Shapiro <cshapiro@google.com> |
Remove trailing whitespace. Change-Id: I95534bb2b88eaf48f2329282041118cd034c812b
|
72e93344b4d1ffc71e9c832ec23de0657e5b04a5 |
|
13-Nov-2009 |
Jean-Baptiste Queru <jbq@google.com> |
eclair snapshot
|
99409883d9c4c0ffb49b070ce307bb33a9dfe9f1 |
|
19-Mar-2009 |
The Android Open Source Project <initial-contribution@android.com> |
auto import //branches/master/...@140412
|
f6c387128427e121477c1b32ad35cdcaa5101ba3 |
|
04-Mar-2009 |
The Android Open Source Project <initial-contribution@android.com> |
auto import from //depot/cupcake/@135843
|
f72d5de56a522ac3be03873bdde26f23a5eeeb3c |
|
04-Mar-2009 |
The Android Open Source Project <initial-contribution@android.com> |
auto import from //depot/cupcake/@135843
|
2ad60cfc28e14ee8f0bb038720836a4696c478ad |
|
21-Oct-2008 |
The Android Open Source Project <initial-contribution@android.com> |
Initial Contribution
|