1393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi# Copyright 2017 The Chromium OS Authors. All rights reserved. 2393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi# Use of this source code is governed by a BSD-style license that can be 3393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi# found in the LICENSE file. 4393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 5393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi"""Wrapper class to store size related information of test results. 6393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi""" 7393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 85e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shiimport contextlib 9393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shiimport json 10393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shiimport os 11393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 12393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shiimport result_info_lib 13393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shiimport utils_lib 14393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 15393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 16393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shiclass ResultInfoError(Exception): 17393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Exception to raise when error occurs in ResultInfo collection.""" 18393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 19393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 20393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shiclass ResultInfo(dict): 21393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """A wrapper class to store result file information. 22393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 23393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi Details of a result include: 24393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi original_size: Original size in bytes of the result, before throttling. 25393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi trimmed_size: Size in bytes after the result is throttled. 26393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi collected_size: Size in bytes of the results collected from the dut. 27393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi files: A list of ResultInfo for the files and sub-directories of the result. 28393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 29393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi The class contains the size information of a result file/directory, and the 30393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi information can be merged if a file was collected multiple times during 31393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi the test. 32393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi For example, `messages` of size 100 bytes was collected before the test 33393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi starts, ResultInfo for this file shall be: 34393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi {'messages': {'/S': 100}} 35393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi Later in the test, the file was collected again when it's size becomes 200 36393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi bytes, the new ResultInfo will be: 37393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi {'messages': {'/S': 200}} 384f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi 394f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi Not that the result infos collected from the dut don't have collected_size 404f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi (/C) set. That's because the collected size in such case is equal to the 414f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi trimmed_size (/T). If the reuslt is not trimmed and /T is not set, the 424f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi value of collected_size can fall back to original_size. The design is to not 434f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi to inject duplicated information in the summary json file, thus reduce the 444f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi size of data needs to be transfered from the dut. 454f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi 46393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi At the end of the test, the file is considered too big, and trimmed down to 47393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 150 bytes, thus the final ResultInfo of the file becomes: 48393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi {'messages': {# The original size is 200 bytes 49393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '/S': 200, 50393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # The total collected size is 300(100+200} bytes 51393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '/C': 300, 52393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # The trimmed size is the final size on disk 53393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '/T': 150} 54393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi From this example, the original size tells us how large the file was. 55393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi The collected size tells us how much data was transfered from dut to drone 56393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi to get this file. And the trimmed size shows the final size of the file when 57393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi the test is finished and the results are throttled again on the server side. 58393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 59393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi The class is a wrapper of dictionary. The properties are all keyvals in a 60393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi dictionary. For example, an instance of ResultInfo can have following 61393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi dictionary value: 62393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi {'debug': { 63393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Original size of the debug folder is 1000 bytes. 64393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '/S': 1000, 65393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # The debug folder was throttled and the size is reduced to 500 66393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # bytes. 67393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '/T': 500, 68393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # collected_size ('/C') can be ignored, its value falls back to 69393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # trimmed_size ('/T'). If trimmed_size is not set, its value falls 70393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # back to original_size ('S') 71393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 72393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Sub-files and sub-directories are included in a list of '/D''s 73393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # value. 74393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # In this example, debug folder has a file `file1`, whose original 75393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # size is 1000 bytes, which is trimmed down to 500 bytes. 76393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '/D': [ 77393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi {'file1': { 78393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '/S': 1000, 79393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '/T': 500, 80393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi } 81393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi } 82393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi ] 83393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi } 84393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi } 85393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 86393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 87393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def __init__(self, parent_dir, name=None, parent_result_info=None, 88393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi original_info=None): 89393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Initialize a collection of size information for a given result path. 90393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 91393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi A ResultInfo object can be initialized in two ways: 92393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 1. Create from a physical file, which reads the size from the file. 93a27022cefc4f3e4e498c99609f9b060b16d07dcaDan Shi In this case, `name` value should be given, and `original_info` 94a27022cefc4f3e4e498c99609f9b060b16d07dcaDan Shi should not be set. 95393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 2. Create from previously collected information, i.e., a dictionary 96393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi deserialized from persisted json file. In this case, `original_info` 97393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi should be given, and `name` should not be set. 98393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 99393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param parent_dir: Path to the parent directory. 100393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param name: Name of the result file or directory. 101393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param parent_result_info: A ResultInfo object for the parent directory. 102393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param original_info: A dictionary of the result's size information. 103393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi This is retrieved from the previously serialized json string. 104393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi For example: {'file_name': 105393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi {'/S': 100, '/T': 50} 106393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi } 107393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi which means a file's original size is 100 bytes, and trimmed 108393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi down to 50 bytes. This argument is used when the object is 109393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi restored from a json string. 110393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 111393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi super(ResultInfo, self).__init__() 112393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 113393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if name is not None and original_info is not None: 114393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi raise ResultInfoError( 115393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 'Only one of parameter `name` and `original_info` can be ' 116393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 'set.') 117393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 118393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # _initialized is a flag to indicating the object is in constructor. 119393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # It can be used to block any size update to make restoring from json 120393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # string faster. For example, if file_details has sub-directories, 121393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # all sub-directories will be added to this class recursively, blocking 122393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # the size updates can reduce unnecessary calculations. 123393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._initialized = False 124393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._parent_result_info = parent_result_info 125393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 126393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if original_info is None: 127393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._init_from_file(parent_dir, name) 128393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi else: 129393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._init_with_original_info(parent_dir, original_info) 130393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 131393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Size of bytes collected in an overwritten or removed directory. 132393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._previous_collected_size = 0 133393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._initialized = True 134393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 135393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def _init_from_file(self, parent_dir, name): 136393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Initialize with the physical file. 137393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 138393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param parent_dir: Path to the parent directory. 139393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param name: Name of the result file or directory. 140393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 141393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi assert name != None 142393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._name = name 143393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 144393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Dictionary to store details of the given path is set to a keyval of 145132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi # the wrapper class. Save the dictionary to an attribute for faster 146132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi # access. 147132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi self._details = {} 148132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi self[self.name] = self._details 149393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 150393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # rstrip is to remove / when name is ROOT_DIR (''). 151393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._path = os.path.join(parent_dir, self.name).rstrip(os.sep) 152393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._is_dir = os.path.isdir(self._path) 153393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 154393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self.is_dir: 155393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # The value of key utils_lib.DIRS is a list of ResultInfo objects. 156393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.details[utils_lib.DIRS] = [] 157393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 158393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Set original size to be the physical size if file details are not 159393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # given and the path is for a file. 160393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self.is_dir: 161393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Set directory size to 0, it will be updated later after its 162393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # sub-directories are added. 163393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.original_size = 0 164393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi else: 165393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.original_size = self.size 166393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 167393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def _init_with_original_info(self, parent_dir, original_info): 168393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Initialize with pre-collected information. 169393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 170393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param parent_dir: Path to the parent directory. 171393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param original_info: A dictionary of the result's size information. 172393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi This is retrieved from the previously serialized json string. 173393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi For example: {'file_name': 174393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi {'/S': 100, '/T': 50} 175393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi } 176393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi which means a file's original size is 100 bytes, and trimmed 177393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi down to 50 bytes. This argument is used when the object is 178393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi restored from a json string. 179393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 180393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi assert original_info 181393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # The result information dictionary has only 1 key, which is the file or 182393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # directory name. 183393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._name = original_info.keys()[0] 184393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 185393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Dictionary to store details of the given path is set to a keyval of 186132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi # the wrapper class. Save the dictionary to an attribute for faster 187132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi # access. 188132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi self._details = {} 189132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi self[self.name] = self._details 190393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 191393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # rstrip is to remove / when name is ROOT_DIR (''). 192393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._path = os.path.join(parent_dir, self.name).rstrip(os.sep) 193393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 194393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._is_dir = utils_lib.DIRS in original_info[self.name] 195393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 196393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self.is_dir: 197393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # The value of key utils_lib.DIRS is a list of ResultInfo objects. 198393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.details[utils_lib.DIRS] = [] 199393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 200393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # This is restoring ResultInfo from a json string. 201393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.original_size = original_info[self.name][ 202393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi utils_lib.ORIGINAL_SIZE_BYTES] 203393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if utils_lib.TRIMMED_SIZE_BYTES in original_info[self.name]: 204393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.trimmed_size = original_info[self.name][ 205393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi utils_lib.TRIMMED_SIZE_BYTES] 206393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self.is_dir: 20783a60bbd061ab37e0e6791563644c9efb8064fb3Dan Shi dirs = original_info[self.name][utils_lib.DIRS] 20883a60bbd061ab37e0e6791563644c9efb8064fb3Dan Shi # TODO: Remove this conversion after R62 is in stable channel. 20983a60bbd061ab37e0e6791563644c9efb8064fb3Dan Shi if isinstance(dirs, dict): 21083a60bbd061ab37e0e6791563644c9efb8064fb3Dan Shi # The summary is generated from older format which stores sub- 21183a60bbd061ab37e0e6791563644c9efb8064fb3Dan Shi # directories in a dictionary, rather than a list. Convert the 21283a60bbd061ab37e0e6791563644c9efb8064fb3Dan Shi # data in old format to a list of dictionary. 21383a60bbd061ab37e0e6791563644c9efb8064fb3Dan Shi dirs = [{dir_name: dirs[dir_name]} for dir_name in dirs] 21483a60bbd061ab37e0e6791563644c9efb8064fb3Dan Shi for sub_file in dirs: 215393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.add_file(None, sub_file) 216393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 2175e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi @contextlib.contextmanager 2185e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi def disable_updating_parent_size_info(self): 2195e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi """Disable recursive calls to update parent result_info's sizes. 2205e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi 2215e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi This context manager allows removing sub-directories to run faster 2225e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi without triggering recursive calls to update parent result_info's sizes. 2235e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi """ 2245e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi old_value = self._initialized 2255e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi self._initialized = False 2265e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi try: 2275e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi yield 2285e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi finally: 2295e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi self._initialized = old_value 2305e8cde91cd0e40044627bc2241de0ff905e0a7b3Dan Shi 231132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi def update_dir_original_size(self): 232132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi """Update all directories' original size information. 233132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi """ 234132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi for f in [f for f in self.files if f.is_dir]: 235132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi f.update_dir_original_size() 236132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi self.update_original_size(skip_parent_update=True) 237132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi 238393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @staticmethod 239393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def build_from_path(parent_dir, 240393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi name=utils_lib.ROOT_DIR, 241393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi parent_result_info=None, top_dir=None, 242393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi all_dirs=None): 243393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Get the ResultInfo for the given path. 244393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 245393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param parent_dir: The parent directory of the given file. 246393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param name: Name of the result file or directory. 247393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param parent_result_info: A ResultInfo instance for the parent 248393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi directory. 249393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param top_dir: The top directory to collect ResultInfo. This is to 250393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi check if a directory is a subdir of the original directory to 251393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi collect summary. 252393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param all_dirs: A set of paths that have been collected. This is to 253393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi prevent infinite recursive call caused by symlink. 254393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 255393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @return: A ResultInfo instance containing the directory summary. 256393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 257132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi is_top_level = top_dir is None 258393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi top_dir = top_dir or parent_dir 259393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi all_dirs = all_dirs or set() 260393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 2614f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi # If the given parent_dir is a file and name is ROOT_DIR, that means 2624f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi # the ResultInfo is for a single file with root directory of the default 2634f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi # ROOT_DIR. 2644f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi if not os.path.isdir(parent_dir) and name == utils_lib.ROOT_DIR: 2654f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi root_dir = os.path.dirname(parent_dir) 2664f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi dir_info = ResultInfo(parent_dir=root_dir, 2674f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi name=utils_lib.ROOT_DIR) 2684f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi dir_info.add_file(os.path.basename(parent_dir)) 2694f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi return dir_info 2704f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi 271393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi dir_info = ResultInfo(parent_dir=parent_dir, 272393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi name=name, 273393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi parent_result_info=parent_result_info) 2744f8c0244d0ce557488a7ddbb29f28a2719a2523fDan Shi 275393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi path = os.path.join(parent_dir, name) 276393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if os.path.isdir(path): 277393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi real_path = os.path.realpath(path) 278393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # The assumption here is that results are copied back to drone by 279393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # copying the symlink, not the content, which is true with currently 280393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # used rsync in cros_host.get_file call. 281393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Skip scanning the child folders if any of following condition is 282393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # true: 283393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # 1. The directory is a symlink and link to a folder under `top_dir` 284393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # 2. The directory was scanned already. 285393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if ((os.path.islink(path) and real_path.startswith(top_dir)) or 286393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi real_path in all_dirs): 287393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return dir_info 288393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi all_dirs.add(real_path) 289393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi for f in sorted(os.listdir(path)): 290393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi dir_info.files.append(ResultInfo.build_from_path( 291393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi parent_dir=path, 292393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi name=f, 293393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi parent_result_info=dir_info, 294393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi top_dir=top_dir, 295393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi all_dirs=all_dirs)) 296132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi 297132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi # Update all directory's original size at the end of the tree building. 298132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi if is_top_level: 299132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi dir_info.update_dir_original_size() 300393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 301393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return dir_info 302393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 303393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 304393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def details(self): 305393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Get the details of the result. 306393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 307393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @return: A dictionary of size and sub-directory information. 308393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 309132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi return self._details 310393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 311393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 312393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def is_dir(self): 313393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Get if the result is a directory. 314393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 315393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return self._is_dir 316393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 317393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 318393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def name(self): 319393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Name of the result. 320393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 321393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return self._name 322393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 323393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 324393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def path(self): 325393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Full path to the result. 326393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 327393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return self._path 328393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 329393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 330393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def files(self): 331393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """All files or sub-directories of the result. 332393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 333393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @return: A list of ResultInfo objects. 334393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @raise ResultInfoError: If the result is not a directory. 335393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 336393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if not self.is_dir: 337393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi raise ResultInfoError('%s is not a directory.' % self.path) 338393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return self.details[utils_lib.DIRS] 339393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 340393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 341393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def size(self): 342393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Physical size in bytes for the result file. 343393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 344393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @raise ResultInfoError: If the result is a directory. 345393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 346393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self.is_dir: 347393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi raise ResultInfoError( 348393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '`size` property does not support directory. Try to use ' 349393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '`original_size` property instead.') 350393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return result_info_lib.get_file_size(self._path) 351393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 352393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 353393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def original_size(self): 354393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """The original size in bytes of the result before it's throttled. 355393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 356393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return self.details[utils_lib.ORIGINAL_SIZE_BYTES] 357393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 358393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @original_size.setter 359393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def original_size(self, value): 360393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Set the original size in bytes of the result. 361393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 362393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param value: The original size in bytes of the result. 363393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 364393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.details[utils_lib.ORIGINAL_SIZE_BYTES] = value 365393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Update the size of parent result infos if the object is already 366393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # initialized. 367393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self._initialized and self._parent_result_info is not None: 368393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._parent_result_info.update_original_size() 369393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 370393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 371393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def trimmed_size(self): 372393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """The size in bytes of the result after it's throttled. 373393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 374393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return self.details.get(utils_lib.TRIMMED_SIZE_BYTES, 375393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.original_size) 376393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 377393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @trimmed_size.setter 378393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def trimmed_size(self, value): 379393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Set the trimmed size in bytes of the result. 380393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 381393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param value: The trimmed size in bytes of the result. 382393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 383393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.details[utils_lib.TRIMMED_SIZE_BYTES] = value 384393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Update the size of parent result infos if the object is already 385393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # initialized. 386393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self._initialized and self._parent_result_info is not None: 387393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._parent_result_info.update_trimmed_size() 388393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 389393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 390393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def collected_size(self): 391393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """The collected size in bytes of the result. 392393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 393393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi The file is throttled on the dut, so the number of bytes collected from 394393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi dut is default to the trimmed_size. If a file is modified between 395393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi multiple result collections and is collected multiple times during the 396393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi test run, the collected_size will be the sum of the multiple 397393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi collections. Therefore, its value will be greater than the trimmed_size 398393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi of the last copy. 399393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 400393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return self.details.get(utils_lib.COLLECTED_SIZE_BYTES, 401393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.trimmed_size) 402393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 403393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @collected_size.setter 404393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def collected_size(self, value): 405393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Set the collected size in bytes of the result. 406393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 407393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param value: The collected size in bytes of the result. 408393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 409393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.details[utils_lib.COLLECTED_SIZE_BYTES] = value 410393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Update the size of parent result infos if the object is already 411393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # initialized. 412393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self._initialized and self._parent_result_info is not None: 413393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._parent_result_info.update_collected_size() 414393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 415393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @property 416393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def is_collected_size_recorded(self): 417393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Flag to indicate if the result has collected size set. 418393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 419393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi This flag is used to avoid unnecessary entry in result details, as the 420393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi default value of collected size is the trimmed size. Removing the 421393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi redundant information helps to reduce the size of the json file. 422393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 423393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return utils_lib.COLLECTED_SIZE_BYTES in self.details 424393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 4254b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi @property 4264b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi def parent_result_info(self): 4274b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi """The result info of the parent directory. 4284b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi """ 4294b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi return self._parent_result_info 4304b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi 4314b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi def add_file(self, name, original_info=None): 432393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Add a file to the result. 433393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 434393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param name: Name of the file. 435393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param original_info: A dictionary of the file's size and sub-directory 436393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi information. 437393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 438393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.details[utils_lib.DIRS].append( 439393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi ResultInfo(parent_dir=self._path, 440393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi name=name, 441393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi parent_result_info=self, 442393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi original_info=original_info)) 443393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # After a new ResultInfo is added, update the sizes if the object is 444393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # already initialized. 445393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self._initialized: 446393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.update_sizes() 447393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 448393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def remove_file(self, name): 449393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Remove a file with the given name from the result. 450393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 451393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param name: Name of the file to be removed. 452393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 453393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.files.remove(self.get_file(name)) 4544b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi # After a new ResultInfo is removed, update the sizes if the object is 4554b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi # already initialized. 4564b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi if self._initialized: 4574b691e55808ec03bd55cbf4000adc98d4fb24dcbDan Shi self.update_sizes() 458393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 459393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def get_file_names(self): 460393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Get a set of all the files under the result. 461393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 462393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return set([f.keys()[0] for f in self.files]) 463393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 464393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def get_file(self, name): 465393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Get a file with the given name under the result. 466393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 467393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param name: Name of the file. 468393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @return: A ResultInfo object of the file. 469393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @raise ResultInfoError: If the result is not a directory, or the file 470393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi with the given name is not found. 471393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 472393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if not self.is_dir: 473393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi raise ResultInfoError('%s is not a directory. Can\'t locate file ' 474393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi '%s' % (self.path, name)) 475393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi for file_info in self.files: 476393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if file_info.name == name: 477393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return file_info 478393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi raise ResultInfoError('Can\'t locate file %s in directory %s' % 479393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi (name, self.path)) 480393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 481393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def convert_to_dir(self): 482393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Convert the result file to a directory. 483393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 484393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi This happens when a result file was overwritten by a directory. The 485393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi conversion will reset the details of this result to be a directory, 486393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi and save the collected_size to attribute `_previous_collected_size`, 487393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi so it can be counted when merging multiple result infos. 488393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 489393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @raise ResultInfoError: If the result is already a directory. 490393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 491393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self.is_dir: 492393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi raise ResultInfoError('%s is already a directory.' % self.path) 493393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # The size that's collected before the file was replaced as a directory. 494393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi collected_size = self.collected_size 495393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._is_dir = True 496393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.details[utils_lib.DIRS] = [] 497393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.original_size = 0 498393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.trimmed_size = 0 499393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._previous_collected_size = collected_size 500393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.collected_size = collected_size 501393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 502132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi def update_original_size(self, skip_parent_update=False): 503393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Update the original size of the result and trigger its parent to 504393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi update. 505132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi 506132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi @param skip_parent_update: True to skip updating parent directory's 507132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi original size. Default is set to False. 508393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 509393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self.is_dir: 510393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.original_size = sum([ 511393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi f.original_size for f in self.files]) 512393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi elif self.original_size is None: 513393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Only set original_size if it's not initialized yet. 514393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.orginal_size = self.size 515393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 516393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Update the size of parent result infos. 517132c7a9b1075acf9c120a84cd24ea12ef05a9e33Dan Shi if not skip_parent_update and self._parent_result_info is not None: 518393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._parent_result_info.update_original_size() 519393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 520393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def update_trimmed_size(self): 521393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Update the trimmed size of the result and trigger its parent to 522393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi update. 523393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 524393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self.is_dir: 525393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_trimmed_size = sum([f.trimmed_size for f in self.files]) 526393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi else: 527393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_trimmed_size = self.size 528393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 529393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Only set trimmed_size if the value is changed or different from the 530393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # original size. 531393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if (new_trimmed_size != self.original_size or 532393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_trimmed_size != self.trimmed_size): 533393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.trimmed_size = new_trimmed_size 534393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 535393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Update the size of parent result infos. 536393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self._parent_result_info is not None: 537393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._parent_result_info.update_trimmed_size() 538393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 539393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def update_collected_size(self): 540393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Update the collected size of the result and trigger its parent to 541393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi update. 542393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 543393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self.is_dir: 544393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_collected_size = ( 545393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._previous_collected_size + 546393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi sum([f.collected_size for f in self.files])) 547393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi else: 548393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_collected_size = self.size 549393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 550393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Only set collected_size if the value is changed or different from the 551393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # trimmed size or existing collected size. 552393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if (new_collected_size != self.trimmed_size or 553393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_collected_size != self.collected_size): 554393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.collected_size = new_collected_size 555393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 556393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Update the size of parent result infos. 557393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if self._parent_result_info is not None: 558393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._parent_result_info.update_collected_size() 559393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 560393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def update_sizes(self): 561393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Update all sizes information of the result. 562393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 563393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.update_original_size() 564393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.update_trimmed_size() 565393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self.update_collected_size() 566393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 56711e350681359d70b15703190c359047aceb4e81bDan Shi def set_parent_result_info(self, parent_result_info, update_sizes=True): 568393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Set the parent result info. 569393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 570393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi It's used when a ResultInfo object is moved to a different file 571393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi structure. 572393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 573393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param parent_result_info: A ResultInfo object for the parent directory. 57411e350681359d70b15703190c359047aceb4e81bDan Shi @param update_sizes: True to update the parent's size information. Set 57511e350681359d70b15703190c359047aceb4e81bDan Shi it to False to delay the update for better performance. 576393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 577393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._parent_result_info = parent_result_info 578393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # As the parent reference changed, update all sizes of the parent. 57911e350681359d70b15703190c359047aceb4e81bDan Shi if parent_result_info and update_sizes: 580393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi self._parent_result_info.update_sizes() 581393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 582393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi def merge(self, new_info, is_final=False): 583393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Merge a ResultInfo instance to the current one. 584393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 585393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi Update the old directory's ResultInfo with the new one. Also calculate 586393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi the total size of results collected from the client side based on the 587393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi difference between the two ResultInfo. 588393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 589393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi When merging with newer collected results, any results not existing in 590393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi the new ResultInfo or files with size different from the newer files 591393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi collected are considered as extra results collected or overwritten by 592393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi the new results. 593393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi Therefore, the size of the collected result should include such files, 594393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi and the collected size can be larger than trimmed size. 595393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi As an example: 596393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi current: {'file1': {TRIMMED_SIZE_BYTES: 1024, 597393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi ORIGINAL_SIZE_BYTES: 1024, 598393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi COLLECTED_SIZE_BYTES: 1024}} 599393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi This means a result `file1` of original size 1KB was collected with size 600393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi of 1KB byte. 601393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_info: {'file1': {TRIMMED_SIZE_BYTES: 1024, 602393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi ORIGINAL_SIZE_BYTES: 2048, 603393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi COLLECTED_SIZE_BYTES: 1024}} 604393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi This means a result `file1` of 2KB was trimmed down to 1KB and was 605393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi collected with size of 1KB byte. 606393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi Note that the second result collection has an updated result `file1` 607393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi (because of the different ORIGINAL_SIZE_BYTES), and it needs to be 608393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi rsync-ed to the drone. Therefore, the merged ResultInfo will be: 609393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi {'file1': {TRIMMED_SIZE_BYTES: 1024, 610393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi ORIGINAL_SIZE_BYTES: 2048, 611393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi COLLECTED_SIZE_BYTES: 2048}} 612393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi Note that: 613393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi * TRIMMED_SIZE_BYTES is still at 1KB, which reflects the actual size of 614393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi the file be collected. 615393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi * ORIGINAL_SIZE_BYTES is updated to 2KB, which is the size of the file 616393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi in the new result `file1`. 617393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi * COLLECTED_SIZE_BYTES is 2KB because rsync will copy `file1` twice as 618393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi it's changed. 619393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 620393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi The only exception is that the new ResultInfo's ORIGINAL_SIZE_BYTES is 621393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi the same as the current ResultInfo's TRIMMED_SIZE_BYTES. That means the 622393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi file was trimmed in the current ResultInfo and the new ResultInfo is 623393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi collecting the trimmed file. Therefore, the merged summary will keep the 624393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi data in the current ResultInfo. 625393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 626393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param new_info: New ResultInfo to be merged into the current one. 627393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param is_final: True if new_info is built from the final result folder. 628393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi Default is set to False. 629393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 630393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_files = new_info.get_file_names() 631393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_files = self.get_file_names() 63211e350681359d70b15703190c359047aceb4e81bDan Shi # A flag to indicate if the sizes need to be updated. It's required when 63311e350681359d70b15703190c359047aceb4e81bDan Shi # child result_info is added to `self`. 63411e350681359d70b15703190c359047aceb4e81bDan Shi update_sizes_pending = False 635393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi for name in new_files: 636393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_file = new_info.get_file(name) 637393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if not name in old_files: 638393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # A file/dir exists in new client dir, but not in the old one, 639393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # which means that the file or a directory is newly collected. 64011e350681359d70b15703190c359047aceb4e81bDan Shi self.files.append(new_file) 64111e350681359d70b15703190c359047aceb4e81bDan Shi # Once parent_result_info is changed, new_file object will no 64211e350681359d70b15703190c359047aceb4e81bDan Shi # longer associated with `new_info` object. 64311e350681359d70b15703190c359047aceb4e81bDan Shi new_file.set_parent_result_info(self, update_sizes=False) 64411e350681359d70b15703190c359047aceb4e81bDan Shi update_sizes_pending = True 645393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi elif new_file.is_dir: 646393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # `name` is a directory in the new ResultInfo, try to merge it 647393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # with the current ResultInfo. 648393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_file = self.get_file(name) 649393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 650393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if not old_file.is_dir: 651393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # If `name` is a file in the current ResultInfo but a 652393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # directory in new ResultInfo, the file in the current 653393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # ResultInfo will be overwritten by the new directory by 654393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # rsync. Therefore, force it to be an empty directory in 655393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # the current ResultInfo, so that the new directory can be 656393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # merged. 657393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_file.convert_to_dir() 658393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 659393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_file.merge(new_file, is_final) 660393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi else: 661393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_file = self.get_file(name) 662393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 663393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # If `name` is a directory in the current ResultInfo, but a file 664393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # in the new ResultInfo, rsync will fail to copy the file as it 665393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # can't overwrite an directory. Therefore, skip the merge. 666393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if old_file.is_dir: 667393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi continue 668393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 669393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_size = new_file.original_size 670393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_size = old_file.original_size 671393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_trimmed_size = new_file.trimmed_size 672393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_trimmed_size = old_file.trimmed_size 673393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 674393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Keep current information if the sizes are not changed. 675393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if (new_size == old_size and 676393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_trimmed_size == old_trimmed_size): 677393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi continue 678393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 679393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Keep current information if the newer size is the same as the 680393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # current trimmed size, and the file is not trimmed in new 681393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # ResultInfo. That means the file was trimmed earlier and stays 682393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # the same when collecting the information again. 683393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if (new_size == old_trimmed_size and 684393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_size == new_trimmed_size): 685393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi continue 686393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 687393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # If the file is merged from the final result folder to an older 688393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # ResultInfo, it's not considered to be trimmed if the size is 689393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # not changed. The reason is that the file on the server side 690393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # does not have the info of its original size. 691393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if is_final and new_trimmed_size == old_trimmed_size: 692393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi continue 693393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 694393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # `name` is a file, and both the original_size and trimmed_size 695393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # are changed, that means the file is overwritten, so increment 696393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # the collected_size. 697393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Before trimming is implemented, collected_size is the 698393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # value of original_size. 699393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_collected_size = new_file.collected_size 700393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_collected_size = old_file.collected_size 701393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 702393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_file.collected_size = ( 703393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_collected_size + old_collected_size) 704393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Only set trimmed_size if one of the following two conditions 705393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # are true: 706393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # 1. In the new summary the file's trimmed size is different 707393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # from the original size, which means the file was trimmed 708393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # in the new summary. 709393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # 2. The original size in the new summary equals the trimmed 710393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # size in the old summary, which means the file was trimmed 711393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # again in the new summary. 712393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi if (new_size == old_trimmed_size or 713393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi new_size != new_trimmed_size): 714393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_file.trimmed_size = new_file.trimmed_size 715393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi old_file.original_size = new_size 716393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 71711e350681359d70b15703190c359047aceb4e81bDan Shi if update_sizes_pending: 71811e350681359d70b15703190c359047aceb4e81bDan Shi self.update_sizes() 71911e350681359d70b15703190c359047aceb4e81bDan Shi 720393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 721393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi# An empty directory, used to compare with a ResultInfo. 722393fc8c903d61bad72ad0ab13d56955ac2888912Dan ShiEMPTY = ResultInfo(parent_dir='', 723393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi original_info={'': {utils_lib.ORIGINAL_SIZE_BYTES: 0, 724393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi utils_lib.DIRS: []}}) 725393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 726393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 727393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shidef save_summary(summary, json_file): 728393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Save the given directory summary to a file. 729393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 730393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param summary: A ResultInfo object for a result directory. 731393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param json_file: Path to a json file to save to. 732393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 733393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi with open(json_file, 'w') as f: 734393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi json.dump(summary, f) 735393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 736393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 737393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shidef load_summary_json_file(json_file): 738393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """Load result info from the given json_file. 739393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 740393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @param json_file: Path to a json file containing a directory summary. 741393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi @return: A ResultInfo object containing the directory summary. 742393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi """ 743393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi with open(json_file, 'r') as f: 744393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi summary = json.load(f) 745393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi 746393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi # Convert summary to ResultInfo objects 747393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi result_dir = os.path.dirname(json_file) 748393fc8c903d61bad72ad0ab13d56955ac2888912Dan Shi return ResultInfo(parent_dir=result_dir, original_info=summary) 749