| fd48cf4fed0be0abdb77b56a377836b1be0f7257 |
|
30-Jan-2018 |
Brennan Saeta <saeta@google.com> |
GCS: Add client-side throttle.
This throttle is loosely based on a leaky bucket configuration that captures
both the cost of the requests as well as the bandwidth consumed into a single
token value.
PiperOrigin-RevId: 183740223
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| b9494ce8990cab65a20f3d5110f4c2c4402342be |
|
26-Jan-2018 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Remove dead code
PiperOrigin-RevId: 183450369
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 9fc2c8ebc8adcbca10c9850a54f913f5e731429f |
|
23-Jan-2018 |
Frank Chen <frankchn@google.com> |
Add functionality to include an optional additional header to be sent on every request when using the GCS file system.
PiperOrigin-RevId: 182855525
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 8aab9b2197efb90d58e86b18b614cd0e63dc56a1 |
|
12-Jan-2018 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Reduce memory allocations in GCS client
This removes some unnecessary transient allocations from the GCS client code, by passing pointers to data directly to the JSON parsing library, rather than copying all of the JSON data to a temporary 'string' object.
Also converted some unnecessarily-general comparisons against Json::Value::null to calls to Value::isNull().
Also changes several parameters from "const string&" to "const char*", in order to avoid unnecessary intermediate allocations. Json::Value::get() has an overload which takes "const char*". Ideally, the JSON library would use string_view or StringPiece, but that's an open source project and so modifying that dependency is out of scope for this change.
PiperOrigin-RevId: 181693172
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| c9ed9bf846c6c8e8566082ce4ac201a529c23355 |
|
05-Jan-2018 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Add a FlushCaches() method to the FileSystem interface, and provide an implementation for GcsFileSystem.
PiperOrigin-RevId: 180873963
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| b639608a6da140f720636582022a575d7c8a7650 |
|
04-Jan-2018 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Fix bug causing OOM failure. An output parameter was not assigned in the failure path of GcsFileSystem::LoadBufferFromGCS(). The fix always zeroes the output parameter, and it also changes the consumer of this API so that the output value is only used when the return Status == OK.
PiperOrigin-RevId: 180847708
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 4edaeba3b14e706740ef4160afa257de565ffd6e |
|
21-Dec-2017 |
Anna R <annarev@google.com> |
Automated g4 rollback of changelist 179729641
PiperOrigin-RevId: 179742670
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| d69515a21cae5bb002ec75783ef973ebf253fdec |
|
20-Dec-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Add the capability of verbose logging for HTTP requests to the TensorFlow GCS client.
This change allows the GCS client within TensorFlow to log verbose information about HTTP requests to the standard Google logger. (Actually, to the TensorFlow variant of the standard Google logger.)
This capability is disabled by default. It is enabled by setting the environment variable GCS_LOG_HTTP_REQUEST_VERBOSE=1 (or =true).
PiperOrigin-RevId: 179729641
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 17900eb2fa0c144d4899b5867f68c010916448c0 |
|
20-Dec-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Convert Status-based call sequence checks to CHECK-based.
Call sequence requirements are correctness requirements; any violation of a correctness requirement is a code defect / bug. Currently, some code in the TF GCS client uses Status objects to report call sequence violations. This tolerates bugs, allowing them to go undetected and uncorrected.
This change converts many call sequence checks to use CHECK, rather than Status. The result is code that is simpler, and that exposes bugs, rather than protecting them.
PiperOrigin-RevId: 179709047
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 3abc6ef02980b0d5fdb226d10cd513122a74c0ea |
|
19-Dec-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Remove a series of allocations and memcpys from the GCS read path (TF infeed path).
The GCS client uses an intermediate std::vector<char> buffer for all reads. This turns out to have a substantial cost, in allocations and in memcpys. During load testing with a simple tool for measuring GCS throughput (but not running TensorFlow), removing this allocation and memcpy increased throughput by about 8%, and significantly reduced steady-state heap memory used. I have not measured the impact when running the full TensorFlow pipeline, but I expect there to be gains, due to the lower memory utilization and the lower cache demand.
PiperOrigin-RevId: 179581563
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 90e42f3ac8c43474633136af4242dca04b6a1e09 |
|
16-Dec-2017 |
Dandelion Man? <dandelion@google.com> |
Automated g4 rollback of changelist 179260538
PiperOrigin-RevId: 179263865
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 9648f8040a559f6cf9bbe0501ba96f2b2c2864b1 |
|
16-Dec-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Automated g4 rollback of changelist 179258973
PiperOrigin-RevId: 179260538
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| d55f532867a3670d66460c5ee3b774519542adc1 |
|
16-Dec-2017 |
Dandelion Man? <dandelion@google.com> |
Merge changes from github.
PiperOrigin-RevId: 179258973
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 7d8ab0e440cf895dceb31f260ce5ae177b84c05d |
|
13-Dec-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
For many requests, the GCS filesystem client did not provide DNS lookup hints. This change allows all GCS HTTP requests to use the GCS DNS cache. It also simplifies the code, and eliminates a lot of redundant code.
The GCS DNS cache has been simplified and made more general. It is now easy to add more DNS names, simply by adding an entry to the GcsDnsCache::names_ list.
PiperOrigin-RevId: 178830317
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 50784e4ef8d920ed5fe7b3a9d0bafbac073ebb26 |
|
12-Dec-2017 |
Brennan Saeta <saeta@google.com> |
Automated g4 rollback of changelist 177619402
PiperOrigin-RevId: 178800980
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 8167f3587f3953e191dffca388cbdd4a837aa231 |
|
01-Dec-2017 |
Gunhan Gulsoy <gunan@google.com> |
Automated g4 rollback of changelist 177614538
PiperOrigin-RevId: 177619402
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| d8e98c8753e3d89e6d4e989e52e910495c4f64ac |
|
01-Dec-2017 |
Brennan Saeta <saeta@google.com> |
Customize request timeouts for GCS filesystem.
PiperOrigin-RevId: 177614538
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| d2e7a2e4bf295a23d6a2e86aa7e0636f00cc2d75 |
|
29-Nov-2017 |
Brennan Saeta <saeta@google.com> |
Add VLOG-ging to gcs_file_system
PiperOrigin-RevId: 177256727
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| e219aeb542779d90a582ffe16f8602cd1b275b22 |
|
22-Nov-2017 |
Brennan Saeta <saeta@google.com> |
GCS: Perform additional file integrity checks
In order to guard against interrupted reads or other network
problems, we perform additional sanity checks to ensure we
correctly load file blocks from GCS.
PiperOrigin-RevId: 176695887
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 9305349a4a6b6c2d265ac81091f855e5560041c4 |
|
22-Nov-2017 |
Brennan Saeta <saeta@google.com> |
Improve caching behavior of GCS filesystem
On a number of Cloud TPU-related workloads, these changes improve the time to
restore from a checkpoint by ~20x.
PiperOrigin-RevId: 176571963
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| b1d89c147d079ab97356b6a677cbf5ee726313d6 |
|
16-Nov-2017 |
Brennan Saeta <saeta@google.com> |
Support user-space DNS caching for the GCS filesystem.
In some environments, DNS resolution is unreliable. This change adds an optional userspace caching mechanism to radically reduce the amount of DNS queries sent to upstream resolvers.
PiperOrigin-RevId: 175910642
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 83c2da808e96dc5c9c3e80353d1db58a17502bf1 |
|
10-Nov-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Removed StringPiece::set and StringPiece::clear, as they have no absl::string_view equivalents.
This will allow for a more convenient transition to absl::string_view.
Calls to set StringPiece::set and StringPiece::clear were replaced with the StringPiece constructor as follows:
string_piece_foo.set(data, size) => string_piece_foo = StringPiece(data, size)
string_piece_foo.clear() => string_piece_foo = StringPiece()
PiperOrigin-RevId: 175326576
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 60205721e1edd791115f8266b84fdd55070d5f1b |
|
29-Sep-2017 |
Derek Murray <mrry@google.com> |
Split `HttpRequest` into an abstract interface and concrete `CurlHttpRequest`.
This is a step towards implementing an HTTP client for platforms where we do
not build libcurl.
PiperOrigin-RevId: 170426868
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 0440c77935915fce0d9286ce5e7407b31c6f4975 |
|
05-Aug-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Add GcsFileSystem caches for Stat and GetMatchingPaths, with configurable maximum age and cache size. Relevant environment variables:
GCS_STAT_CACHE_MAX_AGE (default 0, indicating that the cache is disabled)
GCS_STAT_CACHE_MAX_ENTRIES (default 0, indicating no limit)
GCS_MATCHING_PATHS_CACHE_MAX_AGE (default 0, indicating that the cache is disabled)
GCS_MATCHING_PATHS_CACHE_MAX_ENTRIES (default 0, indicating no limit)
Max age values for these caches are specified in seconds.
Also changed the default GCS read cache block size to 128 MB, so that the block cache is used by default.
PiperOrigin-RevId: 164341936
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 35b606b976de752ee822ace4e290e253bb9482ac |
|
15-Jul-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
FileBlockCache is now a filesystem-wide shared cache with a configurable maximum size (in total bytes of cached block contents). The default maximum cache size is equal to the block size, and can be overridden by the environment variable GCS_READ_CACHE_MAX_SIZE_MB. In addition, deleting files now flushes them from the block cache, and renaming files flushes both the source and destination files from the block cache.
PiperOrigin-RevId: 162041685
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| a641129c9b0b801bc14da3d87d28777926b2540c |
|
14-Jul-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Adds a file cache to the GCS filesystem, with configurable max staleness for file contents. Maximum staleness is specified in seconds in the GCS_READ_CACHE_MAX_STALENESS environment variable, and defaults to 0 (indicating that staleness is not tolerated in a newly opened file). Staleness is measured from the arrival time of the first block in the file's block cache.
PiperOrigin-RevId: 161897769
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 06b22d5d9f4b820734941e056910f698fe3e2405 |
|
05-Jul-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Add a block cache to the GCS filesystem, with block size defaulted to the readahead buffer size. Additionally make the block size and block count configurable via env variables.
PiperOrigin-RevId: 160977285
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| b45ded11d4d9a8f937004782115d2bbb29b97efd |
|
17-May-2017 |
Alexey Surkov <surkov@google.com> |
Make the readahead buffer size configurable via an env variable.
PiperOrigin-RevId: 156238858
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 2ba27efa12283f5a87ea8d81bd75afc46813bd4d |
|
29-Mar-2017 |
Alexey Surkov <surkov@google.com> |
Support for domain buckets, e.g gs://bucketname.domain-example.com/...
Change: 151516660
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 6a9236e060f552cba5883e586d01c5edfa168b42 |
|
17-Mar-2017 |
Alexey Surkov <surkov@google.com> |
Explicilty retry failures of individual deletes in DeleteRecursively.
Otherwise these failuers aren't currently covered by any retry logic.
Change: 150486764
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| a8cd6ff8f1176ea3e7cfd15276051f98e7476500 |
|
16-Mar-2017 |
Alexey Surkov <surkov@google.com> |
Smarter retry logic for non-idempotent file operations such as RenameFile, DeleteFile or DeleteDir.
Change: 150369708
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 3971c27bd2ea9d18d60df9dade623e352a94e723 |
|
14-Mar-2017 |
Alexey Surkov <surkov@google.com> |
Better logging for errors.
1) for resumable uploads, log the last causing failure in the final AbortedError
2) log all retry attempts
Logging example from the unittest:
W0309 18:00:39.706445 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 0.995024 seconds (attempt 1 out of 10), caused by: Unavailable: Failed.
W0309 18:00:39.706562 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 1.39374 seconds (attempt 2 out of 10), caused by: Unavailable: Failed.
W0309 18:00:39.706605 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 2.63587 seconds (attempt 3 out of 10), caused by: Unavailable: Failed.
W0309 18:00:39.706652 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 4.30139 seconds (attempt 4 out of 10), caused by: Unavailable: Failed.
W0309 18:00:39.706693 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 8.32369 seconds (attempt 5 out of 10), caused by: Unavailable: Failed.
W0309 18:00:39.706751 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 16.4618 seconds (attempt 6 out of 10), caused by: Unavailable: Failed.
W0309 18:00:39.706820 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 32.2621 seconds (attempt 7 out of 10), caused by: Unavailable: Failed.
W0309 18:00:39.706867 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 32.2265 seconds (attempt 8 out of 10), caused by: Unavailable: Failed.
W0309 18:00:39.706903 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 32.8241 seconds (attempt 9 out of 10), caused by: Unavailable: Failed.
W0309 18:00:39.706939 687030 retrying_utils.cc:67] The operation failed and will be automatically retried in 32.4939 seconds (attempt 10 out of 10), caused by: Unavailable: Failed.
Change: 150027572
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| c54417fcf33dc7c52a36b3aa0f138abec5ae5a6b |
|
03-Mar-2017 |
Alexey Surkov <surkov@google.com> |
Increase the number of retries to 10.
Also makes file upload retries use the common retry logic.
Change: 149131497
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| bc225bfaa534acc25047fe844f19edc333b7a76a |
|
14-Feb-2017 |
Peter Hawkins <phawkins@google.com> |
Fix code that ignores tensorflow::Status.
Add a new tensorflow::Status::IgnoreError() method to mark call sites where a Status has been intentionally ignored.
Change: 147402405
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 9306b5c0836fef31ed6d99971dc7e0b192cdeb4b |
|
12-Jan-2017 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Keep track of whether a sync is needed to avoid redundant uploads to GCS.
Change: 144355590
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| f97cd45c6b95c6049244842183732aef8c079f8d |
|
15-Nov-2016 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Fixing a bug related to using a string piece after the inner data is destroyed.
The string piece holds a reference to the internal buffer_ when this class gets destructed the buffer gets destructed but the stringpiece can still be used outside (it is an input parameter). The fix is to first copy into scratch (which is an input parameter) and then put the scratch in the stringpiece (result).
Change: 139236445
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| c7a49f9fededb98a7c57ac6acfccddc43dd2f8ed |
|
15-Nov-2016 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Fixing a bug related to copying to the scratch buffer after read.
Change: 139135051
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 133cf792b23c017adef9c64617bbc5ef809fbe66 |
|
12-Nov-2016 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Changing http_request to accept a vector for its output buffer.
Change: 138941640
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 8023ee2e4c7ed97e6d3dd57046b789554a04c066 |
|
10-Nov-2016 |
Alexey Surkov <surkov@google.com> |
Stop remembering when the readahead buffer reached EOF.
This change reverts a previously added optimization for small files and thus unblocks the scenario used by TensorBoard when the same RandomAccessFile object is read for
fetching new content appended to the file.
Change: 138775790
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 879e0accd1c833771c8058d3eb5f2d4f06f895d4 |
|
04-Nov-2016 |
Jonathan Hseu <jhseu@google.com> |
Change FileExists to return tensorflow::Status.
Also done separately by @llhe at github.com/tensorflow/tensorflow/pull/5370. We needed to do this change internally to fix all callers.
Motivation: The existing FileExists interface doesn't allow callers to distinguish between file not found vs. filesystem errors.
Semantics changes:
- gfile.Exists in Python now throws an exception for filesystem errors. It continues to return true/false if it can accurately determine whether a file exists.
- RecursivelyCreateDir now returns errors for filesystem errors when calling FileExists.
Change: 138224013
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 9113b25790bfc4e2da1876a78d74382f776bf02a |
|
03-Nov-2016 |
Jonathan Hseu <jhseu@google.com> |
Change path functions to operate on URIs rather than paths.
- Move the URI parsing and creation functions to path.h so that we don't
depend on core/platform from core/lib/io.
- Fixes https://github.com/tensorflow/tensorflow/issues/5316
Change: 138023841
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 5db4fbe3bbf744ba72d986b4795896b8759031c1 |
|
02-Nov-2016 |
Alexey Surkov <surkov@google.com> |
Make GetMachingPaths also return directory names.
This is now consistent with what other file systems have.
Change: 137892031
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 07ae2d1d606fa3f8f3fb09bcf65b570b3c606173 |
|
02-Nov-2016 |
Jonathan Hseu <jhseu@google.com> |
Fix issue where we wouldn't descend into subdirectories when listing GCS runs
in TensorBoard.
- Fixes GCS so that it still includes prefixes even if items is empty.
- Change list_directory to use GetChildren() instead of GetMatchingFiles.
GetMatchingFiles in GCS has a quirk where, unlike other filesystems, it
doesn't include directories. There's no obvious fix given the GCS API.
Change: 137885363
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| ce92434fd840086f7a275f41024da93431af71f4 |
|
01-Nov-2016 |
Alexey Surkov <surkov@google.com> |
Make GetChildren never return a self directory marker.
In GCS, a folder 'gs://bucket/path' can contain a zero-length object 'gs://bucket/path/', which primarily exists to make IsDirectory('gs://bucket/path') return 'true'.
Naturally GetChildren returned the directory marker for the specified directory whenever it existed, represented as an empty string, which violates assumptions that people normally have from this functions.
Change: 137878839
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 014da84b30b55c5579231840dd45e260d18b5f04 |
|
25-Oct-2016 |
Alexey Surkov <surkov@google.com> |
Better error forwarding in higher-level methods.
The current implementation masks a few errors which prevents retries and
results in irrelevant error logging.
Also remove the already implied strings::StrCat from errors::* macros calls.
Change: 137097532
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| ffdd64b6aae6f2f70f126fcbfc82ce3bfa8ac79c |
|
18-Oct-2016 |
Alexey Surkov <surkov@google.com> |
Implements DeleteRecursively for GcsFileSystem.
GcsFileSystem uses a more efficient way to list all ojbects in a folder than the default FileSystem implementation.
Change: 136423015
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 345b7328871cadaa729052ad5a62a47c26f1595e |
|
15-Oct-2016 |
Alexey Surkov <surkov@google.com> |
GCS Resumable Upload: use int64 instead of int32 when parsing file offsets.
Change: 136219797
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 5c1821be018d4a626efd0a9cee7844aaa8c69366 |
|
15-Oct-2016 |
Alexey Surkov <surkov@google.com> |
Account for the case when GCS Resumable Upload returns a Range header prefixed with "bytes=".
Change: 136211474
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 01b5ab1bc850ecff5c35ac6f936f3952b81208cd |
|
27-Sep-2016 |
Rohan Jain <rohanj@google.com> |
Move Env::GetMatchingPaths implementation to FileSystem instead.
Change: 134358309
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 8c784c51555db944fbe8c6cff702f7201a5a6463 |
|
27-Sep-2016 |
Alexey Surkov <surkov@google.com> |
More efficient implementation of GetMatchingPaths for GCS.
Change: 134335319
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 929160abd520849f0280c7804a7915699ba3f547 |
|
23-Sep-2016 |
Jonathan Hseu <jhseu@google.com> |
Add a ParseURI function and remove the other URI parsing functions.
ParseURI provides all the functionality used by other filesystems right now, so we're able to remove their custom URI parsing code.
- Use ParseURI for URI parsing in GCS and HDFS.
- Note that this fixes a bug where FileSystem::TranslateName("file://foo/bar")
gives "foo/bar" as the path. "file:///" is the correct prefix in that case.
Change: 134009580
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 937d11ae9bd5c36846389192d7475444abd663fb |
|
22-Sep-2016 |
Alexey Surkov <surkov@google.com> |
Safety check for cases when RenameFile tries to move a large object to a bucket with
a different location or storage class.
The situation when object.rewrite doesn't return done=true can occur when
1) the target bucket is in a different location or storage class
and
2) the object is relatively large
See comments for https://cloud.google.com/storage/docs/json_api/v1/objects/copy for details. This situations shouldn't happen in the TensorFlow scenarios, which typically
rename the object within the same bucket, but it's useful to have a safety
check with a meaningful failure message.
Change: 134004191
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 96db4c0fe3170982a0eac38a45da796f726fd247 |
|
22-Sep-2016 |
Vijay Vasudevan <vrv@google.com> |
Make test_benchmark not use RE2. Currently the pattern is not
ever specified by clients, so remove the use of RE2, but leave
comments explaning what we would do.
Also remove unnecessary includes of regexp.h in other files.
Change: 133913048
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 9a7e30a21238e4714265f8ab63c45cde80d6df27 |
|
15-Sep-2016 |
Alexey Surkov <surkov@google.com> |
Bugfixes:
- set CURLOPT_READFUNCTION even when PUT/POST body is empty.
This seems to be solving the problem with hanging requests when PUT/POST body is empty.
- correctly process directory markers when renaming directories
Change: 133202022
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 16a39e515068d0c6141049ea508cab32a527498c |
|
14-Sep-2016 |
Alexey Surkov <surkov@google.com> |
Implements RenameFile for folders.
Change: 133094766
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 9e1cc5d5a2d0396c8f0e2bc60c8f445ff6d44446 |
|
13-Sep-2016 |
Alexey Surkov <surkov@google.com> |
Switches from simple uploads to resumable uploads in GCS.
According to the GCS docs, simple uploads are only recommended for files below 5Mb:
https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload
This CL starts using the resumable upload API, including the retry logic recommended in the doc:
https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload
Change: 133051424
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 1d4bc7abe7468ae1b039bf832c141d5104deeb55 |
|
12-Sep-2016 |
Alexey Surkov <surkov@google.com> |
Emulate subfolders for GCS to correctly handle directory management operations.
Change: 132893746
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| c2249535e54a68e3784f80ebfcf6caab7fe5a708 |
|
09-Sep-2016 |
Derek Murray <mrry@google.com> |
Replace `FileStatistics::mode` with `FileStatistics::is_directory`.
The output of `stat()` is platform- and filesystem-dependent, so the
current API that exports a file's mode as a bitfield is not
portable. For example, on POSIX-based systems we use `mode_t` and
`S_ISDIR()` to test whether a file is a directory, whereas on Windows
we would would access the `_S_IFDIR` bit of an `st_mode`
bitfield. Note also that the GCS filesystem would set the mode
arbitrarily as 0600, because there is no direct mapping between GCS
file attributes and POSIX modes.
This CL removes the `mode_t mode` field and replaces it with a `bool
is_directory` field, which corresponds to the only use of that field
that I've been able to find, and which can be set as appropriate in
platform-specific code. If and when other attributes of a file
(e.g. permissions) are needed, these can be handled similarly.
Change: 132698567
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 10272d2bb7275a9e7c0b266928d8d9b3763d8f44 |
|
09-Sep-2016 |
Alexey Surkov <surkov@google.com> |
Add basic mutex synchronization to GcsFileSystem and GoogleAuthProvider.
Change: 132623564
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 7a1210bdbdade7210d48db287065ecac950338aa |
|
01-Sep-2016 |
Vincent Vanhoucke <vanhoucke@google.com> |
Fix ~63 ClangTidy - Performance findings in TensorFlow.
Change: 131891101
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 01d291f7c8d6ae9e9ea29a7c689fedc8f31c1c42 |
|
31-Aug-2016 |
Alexey Surkov <surkov@google.com> |
Remember when the readahead buffer reaches EOF.
Previously, if a new request had a range ending after EOF, the readhead buffer got invalidated and a new HTTP request were made.
With this change, the buffer will remember that there is no data left and will still be used to answer such requests.
Change: 131761010
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 6e95ddd1eaa585acf555c7bca5cdb6384425e47e |
|
24-Aug-2016 |
A. Unique TensorFlower <gardener@tensorflow.org> |
GCS File system optimization.
Current implementation allocates new buffer every time it reads buffer size of bytes from the file.
Buffer can be re-allocated many times for the same file.
After this change buffer gets re-used and grows only when needed.
Change: 131182639
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 2c90008e49b78d1b1731aacd6b4e28d8c8e690ab |
|
24-Aug-2016 |
Alexey Surkov <surkov@google.com> |
Implements GcsFileSystem::Stat
Change: 131115272
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 5f4931f2662bf130f34aecab1a8164b7bfa0af57 |
|
20-Aug-2016 |
Alexey Surkov <surkov@google.com> |
More verbose GCS errors.
Change: 130801797
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 2094bb5349c060516907f02975bcdc6e752d407b |
|
13-Jul-2016 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Adding FileSystem::Stat to the core filesystem API. Implemented versions of it for all subclasses except for GCS File System for now.
Change: 127282696
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 26217c20fb0b2766e2c17430340bdacc4f41f5c2 |
|
01-Jul-2016 |
Alexey Surkov <surkov@google.com> |
Add retry logic to all GCS operations.
Also adds a few minor fixes for Google auth:
- support for the no-auth case (public objects will still work)
- less tolerance for almost expired tokens
- better error logging for failed token retrievals
Change: 126449395
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 3854ea49a44de3eb2e3eb225b9e35fbbd3a965d8 |
|
24-Jun-2016 |
Alexey Surkov <surkov@google.com> |
Minor improvements to GcsFileSystem.
- adds support of paths pointing to the root of a bucket, e.g gs://bucket
- adds pagination support for GetChildren
- makes a more optimal HTTP request in GetChildren
Change: 125785863
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 8bf25a491b60d223bba11233de9e62f4b0db17e8 |
|
23-Jun-2016 |
Alexey Surkov <surkov@google.com> |
Add a read-ahead cache to the GCS implementation of RandomAccessFile.
In some cases TensorFlow reads the data via RandomAccessFile in really small
chunks, which doesn't work very efficiently with HTTP requests. Adding a
read-ahead cache significantly boosts the performance.
Change: 125691397
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 209c006578483460e98c114a69d4b9ed6b95efed |
|
17-Jun-2016 |
Vijay Vasudevan <vrv@google.com> |
TensorFlow: minor change functions in Env interface.
Switches to return a std::unique_ptr<> to convey transfer of ownership.
Keeps existing functions temporarily for backwards compatibility.
Change file_system.h interface to switch entirely to the new interface,
change all callers. If this breaks someone in the public, the interface
change should be straightforward.
Change: 125127028
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| 92f44afcfb35ca4e212df7f8e0bfca245707cc06 |
|
10-Jun-2016 |
A. Unique TensorFlower <nobody@tensorflow.org> |
Changing GetChildren to not return names with complete gs:// path.
Change: 124503860
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| d94f70b4380385e7ab4cddb68e42bb53c6e18954 |
|
08-Jun-2016 |
A. Unique TensorFlower <nobody@tensorflow.org> |
Making sure when accessing GCS the object path is url-encoded. According to GCS docs, object path has to be url-encoded.
Change: 124326567
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| c8b59c046895fa5b6d79f73e0b5817330fcfbfc1 |
|
02-Jun-2016 |
A. Unique TensorFlower <nobody@tensorflow.org> |
Update copyright for 3p/tf/core.
Change: 123900938
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| bb465cdc0ed9c3b9b4f031505ea2294375677807 |
|
19-May-2016 |
Alexey Surkov <surkov@google.com> |
Google authentication for GCS file system.
Implements an authentication mechanism based on Application Default Credentials:
https://developers.google.com/identity/protocols/application-default-credentials
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
Change: 122741738
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|
| ed65e69560a8e2d58f7571fc2dcac269b20c260d |
|
12-May-2016 |
A. Unique TensorFlower <nobody@tensorflow.org> |
File system implementation for Google Cloud Storage.
This code implements a file system for file paths starting with gs:// using the HTTP API to Google Cloud Storage. No authentication is implemented yet, so only GCS objects with public access can be used.
Change: 122126085
/external/tensorflow/tensorflow/core/platform/cloud/gcs_file_system.cc
|