1e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# Copyright 2016 The TensorFlow Authors. All Rights Reserved. 2e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# 3e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# Licensed under the Apache License, Version 2.0 (the "License"); 4e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# you may not use this file except in compliance with the License. 5e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# You may obtain a copy of the License at 6e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# 7e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# http://www.apache.org/licenses/LICENSE-2.0 8e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# 9e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# Unless required by applicable law or agreed to in writing, software 10e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# distributed under the License is distributed on an "AS IS" BASIS, 11e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# See the License for the specific language governing permissions and 13e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# limitations under the License. 14e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai# ============================================================================== 15e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai"""TensorFlow Debugger (tfdbg) Utilities.""" 16b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai 17e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Caifrom __future__ import absolute_import 18e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Caifrom __future__ import division 19e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Caifrom __future__ import print_function 20e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 21e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Caiimport re 22e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 23e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Caifrom six.moves import xrange # pylint: disable=redefined-builtin 24e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 25e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 26e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Caidef add_debug_tensor_watch(run_options, 27e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai node_name, 28e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai output_slot=0, 29e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_ops="DebugIdentity", 30aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai debug_urls=None, 31e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures=False, 32aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai global_step=-1): 33b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai """Add watch on a `Tensor` to `RunOptions`. 34b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai 35d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai N.B.: 36d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai 1. Under certain circumstances, the `Tensor` may not get actually watched 37d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai (e.g., if the node of the `Tensor` is constant-folded during runtime). 38d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai 2. For debugging purposes, the `parallel_iteration` attribute of all 39d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai `tf.while_loop`s in the graph are set to 1 to prevent any node from 40d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai being executed multiple times concurrently. This change does not affect 41d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai subsequent non-debugged runs of the same `tf.while_loop`s. 42e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 43e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai Args: 44b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai run_options: An instance of `config_pb2.RunOptions` to be modified. 45b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai node_name: (`str`) name of the node to watch. 46b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai output_slot: (`int`) output slot index of the tensor from the watched node. 47b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai debug_ops: (`str` or `list` of `str`) name(s) of the debug op(s). Can be a 48b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai `list` of `str` or a single `str`. The latter case is equivalent to a 49b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai `list` of `str` with only one element. 50bfe56aaafd87115ffbd8479f49a4f267d9a923fbShanqing Cai For debug op types with customizable attributes, each debug op string can 51bfe56aaafd87115ffbd8479f49a4f267d9a923fbShanqing Cai optionally contain a list of attribute names, in the syntax of: 52bfe56aaafd87115ffbd8479f49a4f267d9a923fbShanqing Cai debug_op_name(attr_name_1=attr_value_1;attr_name_2=attr_value_2;...) 53b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai debug_urls: (`str` or `list` of `str`) URL(s) to send debug values to, 54b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai e.g., `file:///tmp/tfdbg_dump_1`, `grpc://localhost:12345`. 55e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures: (`bool`) Whether to tolerate debug op 56e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai creation failures by not throwing exceptions. 57aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai global_step: (`int`) Optional global_step count for this debug tensor 58aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai watch. 59e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai """ 60e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 61db2a81a82c1753154648309f70f6ed3fa8f60837A. Unique TensorFlower watch_opts = run_options.debug_options.debug_tensor_watch_opts 62aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai run_options.debug_options.global_step = global_step 63e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 64e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai watch = watch_opts.add() 65e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai watch.tolerate_debug_op_creation_failures = ( 66e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures) 67e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai watch.node_name = node_name 68e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai watch.output_slot = output_slot 69e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 70e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if isinstance(debug_ops, str): 71e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_ops = [debug_ops] 72e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 73e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai watch.debug_ops.extend(debug_ops) 74e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 75e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if debug_urls: 76e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if isinstance(debug_urls, str): 77e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_urls = [debug_urls] 78e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 79e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai watch.debug_urls.extend(debug_urls) 80e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 81e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 82e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Caidef watch_graph(run_options, 83e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai graph, 84e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_ops="DebugIdentity", 85e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_urls=None, 86e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai node_name_regex_whitelist=None, 87aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai op_type_regex_whitelist=None, 881a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai tensor_dtype_regex_whitelist=None, 89e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures=False, 90aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai global_step=-1): 91b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai """Add debug watches to `RunOptions` for a TensorFlow graph. 92e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 93b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai To watch all `Tensor`s on the graph, let both `node_name_regex_whitelist` 94b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai and `op_type_regex_whitelist` be the default (`None`). 95b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai 96d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai N.B.: 97d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai 1. Under certain circumstances, the `Tensor` may not get actually watched 98d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai (e.g., if the node of the `Tensor` is constant-folded during runtime). 99d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai 2. For debugging purposes, the `parallel_iteration` attribute of all 100d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai `tf.while_loop`s in the graph are set to 1 to prevent any node from 101d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai being executed multiple times concurrently. This change does not affect 102d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai subsequent non-debugged runs of the same `tf.while_loop`s. 103d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai 104e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 105e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai Args: 106b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai run_options: An instance of `config_pb2.RunOptions` to be modified. 107b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai graph: An instance of `ops.Graph`. 108b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai debug_ops: (`str` or `list` of `str`) name(s) of the debug op(s) to use. 109b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai debug_urls: URLs to send debug values to. Can be a list of strings, 110b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai a single string, or None. The case of a single string is equivalent to 111b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai a list consisting of a single string, e.g., `file:///tmp/tfdbg_dump_1`, 112b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai `grpc://localhost:12345`. 113bfe56aaafd87115ffbd8479f49a4f267d9a923fbShanqing Cai For debug op types with customizable attributes, each debug op name string 114bfe56aaafd87115ffbd8479f49a4f267d9a923fbShanqing Cai can optionally contain a list of attribute names, in the syntax of: 115bfe56aaafd87115ffbd8479f49a4f267d9a923fbShanqing Cai debug_op_name(attr_name_1=attr_value_1;attr_name_2=attr_value_2;...) 116b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai node_name_regex_whitelist: Regular-expression whitelist for node_name, 117b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai e.g., `"(weight_[0-9]+|bias_.*)"` 118e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai op_type_regex_whitelist: Regular-expression whitelist for the op type of 119b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai nodes, e.g., `"(Variable|Add)"`. 120b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai If both `node_name_regex_whitelist` and `op_type_regex_whitelist` 121b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai are set, the two filtering operations will occur in a logical `AND` 122b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai relation. In other words, a node will be included if and only if it 123b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai hits both whitelists. 1241f6ed06221949c5a9037f2d81f30208d23f3afc6Andreas Solleder tensor_dtype_regex_whitelist: Regular-expression whitelist for Tensor 1251a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai data type, e.g., `"^int.*"`. 1261a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai This whitelist operates in logical `AND` relations to the two whitelists 1271a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai above. 128e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures: (`bool`) whether debug op creation 129e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai failures (e.g., due to dtype incompatibility) are to be tolerated by not 130e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai throwing exceptions. 131aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai global_step: (`int`) Optional global_step count for this debug tensor 132aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai watch. 133e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai """ 134e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 135e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if isinstance(debug_ops, str): 136e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_ops = [debug_ops] 137e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 1381a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai node_name_pattern = (re.compile(node_name_regex_whitelist) 1391a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai if node_name_regex_whitelist else None) 1401a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai op_type_pattern = (re.compile(op_type_regex_whitelist) 1411a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai if op_type_regex_whitelist else None) 1421a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai tensor_dtype_pattern = (re.compile(tensor_dtype_regex_whitelist) 1431a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai if tensor_dtype_regex_whitelist else None) 144e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 145e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai ops = graph.get_operations() 146e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai for op in ops: 147e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai # Skip nodes without any output tensors. 148e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if not op.outputs: 149e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai continue 150e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 151e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai node_name = op.name 152e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai op_type = op.type 153e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 154e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if node_name_pattern and not node_name_pattern.match(node_name): 155e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai continue 156e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if op_type_pattern and not op_type_pattern.match(op_type): 157e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai continue 158e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 159e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai for slot in xrange(len(op.outputs)): 1601a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai if (tensor_dtype_pattern and 1611a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai not tensor_dtype_pattern.match(op.outputs[slot].dtype.name)): 1621a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai continue 1631a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai 164e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai add_debug_tensor_watch( 165e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai run_options, 166e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai node_name, 167e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai output_slot=slot, 168e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_ops=debug_ops, 169aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai debug_urls=debug_urls, 170e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures=( 171e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures), 172aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai global_step=global_step) 173e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 174e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 175e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Caidef watch_graph_with_blacklists(run_options, 176e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai graph, 177e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_ops="DebugIdentity", 178e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_urls=None, 179e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai node_name_regex_blacklist=None, 180aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai op_type_regex_blacklist=None, 1811a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai tensor_dtype_regex_blacklist=None, 182e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures=False, 183aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai global_step=-1): 184b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai """Add debug tensor watches, blacklisting nodes and op types. 185e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 186b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai This is similar to `watch_graph()`, but the node names and op types are 187e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai blacklisted, instead of whitelisted. 188e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 189d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai N.B.: 190d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai 1. Under certain circumstances, the `Tensor` may not get actually watched 191d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai (e.g., if the node of the `Tensor` is constant-folded during runtime). 192d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai 2. For debugging purposes, the `parallel_iteration` attribute of all 193d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai `tf.while_loop`s in the graph are set to 1 to prevent any node from 194d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai being executed multiple times concurrently. This change does not affect 195d8d0bab6301ea7a4f32eb5cc6b13d1d495aa5e7aShanqing Cai subsequent non-debugged runs of the same `tf.while_loop`s. 196b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai 197e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai Args: 198b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai run_options: An instance of `config_pb2.RunOptions` to be modified. 199b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai graph: An instance of `ops.Graph`. 200b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai debug_ops: (`str` or `list` of `str`) name(s) of the debug op(s) to use. 201bfe56aaafd87115ffbd8479f49a4f267d9a923fbShanqing Cai See the documentation of `watch_graph` for more details. 202f8838fabb0a18ea8ad852718e3c1ded4dcc5917dPatrick Nguyen debug_urls: URL(s) to send debug values to, e.g., 203b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai `file:///tmp/tfdbg_dump_1`, `grpc://localhost:12345`. 204e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai node_name_regex_blacklist: Regular-expression blacklist for node_name. 205b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai This should be a string, e.g., `"(weight_[0-9]+|bias_.*)"`. 206e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai op_type_regex_blacklist: Regular-expression blacklist for the op type of 207b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai nodes, e.g., `"(Variable|Add)"`. 208b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai If both node_name_regex_blacklist and op_type_regex_blacklist 209b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai are set, the two filtering operations will occur in a logical `OR` 210b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai relation. In other words, a node will be excluded if it hits either of 211b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai the two blacklists; a node will be included if and only if it hits 212b04d5686b918ed10234dd6689fef6965b81f575cShanqing Cai neither of the blacklists. 2131f6ed06221949c5a9037f2d81f30208d23f3afc6Andreas Solleder tensor_dtype_regex_blacklist: Regular-expression blacklist for Tensor 2141a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai data type, e.g., `"^int.*"`. 2151a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai This blacklist operates in logical `OR` relations to the two whitelists 2161a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai above. 217e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures: (`bool`) whether debug op creation 218e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai failures (e.g., due to dtype incompatibility) are to be tolerated by not 219e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai throwing exceptions. 220aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai global_step: (`int`) Optional global_step count for this debug tensor 221aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai watch. 222e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai """ 223e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 224e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if isinstance(debug_ops, str): 225e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_ops = [debug_ops] 226e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 2271a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai node_name_pattern = (re.compile(node_name_regex_blacklist) if 2281a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai node_name_regex_blacklist else None) 2291a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai op_type_pattern = (re.compile(op_type_regex_blacklist) if 2301a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai op_type_regex_blacklist else None) 2311a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai tensor_dtype_pattern = (re.compile(tensor_dtype_regex_blacklist) if 2321a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai tensor_dtype_regex_blacklist else None) 233e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 234e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai ops = graph.get_operations() 235e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai for op in ops: 236e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai # Skip nodes without any output tensors. 237e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if not op.outputs: 238e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai continue 239e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 240e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai node_name = op.name 241e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai op_type = op.type 242e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 243e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if node_name_pattern and node_name_pattern.match(node_name): 244e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai continue 245e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai if op_type_pattern and op_type_pattern.match(op_type): 246e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai continue 247e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai 248e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai for slot in xrange(len(op.outputs)): 2491a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai if (tensor_dtype_pattern and 2501a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai tensor_dtype_pattern.match(op.outputs[slot].dtype.name)): 2511a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai continue 2521a833fb731d0daca9a10d55dc9d9f3c4dcaeaa1eShanqing Cai 253e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai add_debug_tensor_watch( 254e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai run_options, 255e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai node_name, 256e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai output_slot=slot, 257e6d5c25269546b3f6f2fa0442df6b5889af84f2cShanqing Cai debug_ops=debug_ops, 258aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai debug_urls=debug_urls, 259e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures=( 260e3d1c7d515d7c7b335a1fd58dde03b7f52409233Shanqing Cai tolerate_debug_op_creation_failures), 261aabc7972b94af5a678550427534d4fba7fda327cShanqing Cai global_step=global_step) 262