| b1f5f433959406c7aad634c05e85ccd62fd06e87 |
|
08-Feb-2018 |
Yong Tang <yong.tang.github@outlook.com> |
Support CopyFile with streaming (#12658)
* Support CopyFile with streaming
This fix tries to address the issue raised in 12641 where
it was not possible to have CopyFile with streaming. The original
implementation copies the whole content of the file to a string
buffer and write to the file. This could be an issue
if the file size is large (than memory of the host).
This fix streams the CopyFile operation.
This fix fixes 12641.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Use sendfile for CopyFile implementation in Linux
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Merge CopyFile for same fs and different fs
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* `sendfile64` -> `sendfile` to fix Android build
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Add sendfile processing for Darwin
This commit adds sendfile processing for OSX Darwin.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Not using sendfile in MacOSX
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Address review feedback
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Remove the size check and test OUT_OF_RANGE instead.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Small fixes
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Rename CopyFile to FileSystemCopyFile to fix Windows build errors
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| c9ed9bf846c6c8e8566082ce4ac201a529c23355 |
|
05-Jan-2018 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Add a FlushCaches() method to the FileSystem interface, and provide an implementation for GcsFileSystem.
PiperOrigin-RevId: 180873963
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 96cb8f886ad84202e363c5a9da56cdbce4eaf408 |
|
08-Mar-2017 |
Benoit Steiner <bsteiner@google.com> |
Started to open source Grappler. First application is the GPU layout optimizer.
Change: 149558284
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| c14d395a7dd96cc622654cd096bc284aa81c9049 |
|
09-Feb-2017 |
Vijay Vasudevan <vrv@google.com> |
Add documentation to file_system.h, much of it borrowed from env.h.
Change: 147075674
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 723c4048790d6f0636f6c1df5f4fb793ef7a4ae6 |
|
21-Jan-2017 |
Suharsh Sivakumar <suharshs@google.com> |
FileSystem directory creation fixes:
- Ensure that CreateDir returns error::ALREADY_EXISTS if the dirname exists.
- Ensure that RecursivelyCreateDirectory ignores error::ALREADY_EXISTS when
creating directories and subdirectories.
Fixes #6974
Change: 145144720
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 879e0accd1c833771c8058d3eb5f2d4f06f895d4 |
|
04-Nov-2016 |
Jonathan Hseu <jhseu@google.com> |
Change FileExists to return tensorflow::Status.
Also done separately by @llhe at github.com/tensorflow/tensorflow/pull/5370. We needed to do this change internally to fix all callers.
Motivation: The existing FileExists interface doesn't allow callers to distinguish between file not found vs. filesystem errors.
Semantics changes:
- gfile.Exists in Python now throws an exception for filesystem errors. It continues to return true/false if it can accurately determine whether a file exists.
- RecursivelyCreateDir now returns errors for filesystem errors when calling FileExists.
Change: 138224013
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 9113b25790bfc4e2da1876a78d74382f776bf02a |
|
03-Nov-2016 |
Jonathan Hseu <jhseu@google.com> |
Change path functions to operate on URIs rather than paths.
- Move the URI parsing and creation functions to path.h so that we don't
depend on core/platform from core/lib/io.
- Fixes https://github.com/tensorflow/tensorflow/issues/5316
Change: 138023841
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 120c3f11f857ce6da7ae90b3eab943896499dca2 |
|
11-Oct-2016 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Small header changes and a Doxygen tweak to shape the C++ API a bit better.
Change: 135747447
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| edaf3b342db4afa1c872da541fb0ac176a4e8ef9 |
|
10-Oct-2016 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Merge changes from github.
Change: 135698415
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 4a3065a1b796e8c5b52987cc3442ea3d529369c1 |
|
30-Sep-2016 |
Rohan Jain <rohanj@google.com> |
Moving implementation of RecursivelyCreateDir and DeleteRecursively down to the FileSystem from the Env so that file systems could override if needed.
Change: 134725784
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| c45200a8e6073ea1902b5a754d957e816bde18df |
|
29-Sep-2016 |
Yuefeng Zhou <yuefengz@google.com> |
Overwrite the target file in HDFS Rename operation; it removes the target file first if it exists.
Change: 134682344
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 01b5ab1bc850ecff5c35ac6f936f3952b81208cd |
|
27-Sep-2016 |
Rohan Jain <rohanj@google.com> |
Move Env::GetMatchingPaths implementation to FileSystem instead.
Change: 134358309
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 8c784c51555db944fbe8c6cff702f7201a5a6463 |
|
27-Sep-2016 |
Alexey Surkov <surkov@google.com> |
More efficient implementation of GetMatchingPaths for GCS.
Change: 134335319
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| d63243106a818a415dd9894f462a5cfc26081aee |
|
23-Sep-2016 |
Jonathan Hseu <jhseu@google.com> |
Fix Env::RecursivelyCreateDir handling for URIs, fixing it for GCS and HDFS.
Change: 134110801
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 929160abd520849f0280c7804a7915699ba3f547 |
|
23-Sep-2016 |
Jonathan Hseu <jhseu@google.com> |
Add a ParseURI function and remove the other URI parsing functions.
ParseURI provides all the functionality used by other filesystems right now, so we're able to remove their custom URI parsing code.
- Use ParseURI for URI parsing in GCS and HDFS.
- Note that this fixes a bug where FileSystem::TranslateName("file://foo/bar")
gives "foo/bar" as the path. "file:///" is the correct prefix in that case.
Change: 134009580
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 8f49b715da73e877174a0decbf3751ef4004282d |
|
21-Sep-2016 |
Derek Murray <mrry@google.com> |
Clean up warnings in tensorflow/core/platform.
Some compilers warn about symbols with no implementations; use
TF_DISALLOW_COPY_AND_ASSIGN instead; also include a missing header to
make the use of `string` vs `tensorflow::string` unambiguous.
Change: 133827071
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 857fc259445fbff9a23ba1723d7f617ff40fbf05 |
|
14-Sep-2016 |
Rohan Jain <rohanj@google.com> |
Adding io::CleanPath in path.h to clean up paths. Also made TranslateName call CleanPath so that all paths are clean before doing file system ops.
Change: 133140241
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 56ad910f5957d3aa2d96eb840e6da36ac4105236 |
|
09-Sep-2016 |
Rohan Jain <rohanj@google.com> |
Redirecting gfile to file_io.
Also, changed the return type of FileIO.readline(), FileIO.readlines(), list_directories() and walk() to be strings rather than bytes.
Change: 132634441
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| bc81b2c0cd7d70db6016a183591f05989f03f822 |
|
02-Aug-2016 |
Rohan Jain <rohanj@google.com> |
Adding support for Stat in the FileIO API.
Change: 129043456
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 2094bb5349c060516907f02975bcdc6e752d407b |
|
13-Jul-2016 |
A. Unique TensorFlower <gardener@tensorflow.org> |
Adding FileSystem::Stat to the core filesystem API. Implemented versions of it for all subclasses except for GCS File System for now.
Change: 127282696
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 209c006578483460e98c114a69d4b9ed6b95efed |
|
17-Jun-2016 |
Vijay Vasudevan <vrv@google.com> |
TensorFlow: minor change functions in Env interface.
Switches to return a std::unique_ptr<> to convey transfer of ownership.
Keeps existing functions temporarily for backwards compatibility.
Change file_system.h interface to switch entirely to the new interface,
change all callers. If this breaks someone in the public, the interface
change should be straightforward.
Change: 125127028
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| c8b59c046895fa5b6d79f73e0b5817330fcfbfc1 |
|
02-Jun-2016 |
A. Unique TensorFlower <nobody@tensorflow.org> |
Update copyright for 3p/tf/core.
Change: 123900938
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 3c280f6fa0e0fcaa3d2cee5d2d8bb7ab3e25319f |
|
18-Apr-2016 |
A. Unique TensorFlower <nobody@tensorflow.org> |
Added a format for saving an inference graph that can be memmapped and an utility to convert a freezed graph into this format.
Change: 120128412
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| f77c9fb707d12f5354a399055b6db5ebd5bc5d1f |
|
09-Apr-2016 |
Manjunath Kudlur <keveman@gmail.com> |
Remove unused mutex.h from file_system.h
Change: 119420831
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 6cdcc5945183070d51b85cba79c089416e55a736 |
|
08-Apr-2016 |
Manjunath Kudlur <keveman@gmail.com> |
Move the FileSystemRegistry inside Env instead of it being global and shared
by all Envs.
Change: 119342847
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| fe5c6ae460e40a92798d0383026ab0ef85026bed |
|
07-Apr-2016 |
Manjunath Kudlur <keveman@gmail.com> |
Implement a file factory mechanism to handle network file systems.
- Env dispatches to a FileSystem interface
- FileSystemFactory is used to look up the correct FileSystem implementation
based on the prefix of the filename
- Provide a registration mechanism to register different factories
Change: 119268846
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 9cc7cf63ae3f7cd38c3bd7b7aac77aa99c7e1951 |
|
07-Apr-2016 |
Manjunath Kudlur <keveman@gmail.com> |
Implement a file factory mechanism to handle network file systems.
- Env dispatches to a FileSystem interface
- FileSystemFactory is used to look up the correct FileSystem implementation
based on the prefix of the filename
- Provide a registration mechanism to register different factories
Change: 119236345
/external/tensorflow/tensorflow/core/platform/file_system.h
|
| 9671b6c17f6d4809cc37c3218e27f885084b6253 |
|
07-Apr-2016 |
Manjunath Kudlur <keveman@gmail.com> |
Implement a file factory mechanism to handle network file systems.
- Env dispatches to a FileSystem interface
- FileSystemFactory is used to look up the correct FileSystem implementation
based on the prefix of the filename
- Provide a registration mechanism to register different factories
Change: 119233977
/external/tensorflow/tensorflow/core/platform/file_system.h
|