public final class AzureFileSystemProvider extends FileSystemProvider
AzureFileSystemProvider
is Azure Storage's implementation of the nio interface on top of Azure Blob
Storage.
Particular care should be taken when working with a remote storage service. This implementation makes no guarantees
on behavior or state should other processes operate on the same data concurrently; file systems from this provider
will assume they have exclusive access to their data and will behave without regard for potential of interfering
applications. Moreover, remote file stores introduce higher latencies. Therefore, additional consideration should be
given to managing concurrency: race conditions are more likely to manifest and network failures occur more frequently
than disk failures. These and other such distributed application scenarios must be considered when working with this
file system. While the AzureFileSystem
will ensure it takes appropriate steps towards robustness and
reliability, the application developer must design around these failure scenarios and have fallback and retry options
available.
The Azure Blob Storage service backing these APIs is not a true FileSystem, nor is it the goal of this implementation
to force Azure Blob Storage to act like a full-fledged file system. Some APIs and scenarios will remain unsupported
indefinitely until they may be sensibly implemented. Other APIs may experience lower performance than is expected
because of the number of network requests needed to ensure correctness. The javadocs for each type and method should
also be read carefully to understand what guarantees are made and how they may differ from the contract defined by
FileSystemProvider
.
The scheme for this provider is "azb"
, and the format of the URI to identify an AzureFileSystem
is
"azb://?endpoint=<endpoing>"
. The endpoint of the Storage account is used to uniquely identify the
filesystem.
An AzureFileSystem
is backed by an account. An AzureFileStore
is backed by a container. Any number of
containers may be specified as file stores upon creation of the file system. When a file system is created,
it will try to retrieve the properties of each container to ensure connection to the account. If any of the
containers does not exist, it will be created. Failure to access or create containers as necessary will result in
an exception and failure to create the file system. Any data existing in the containers will be preserved and
accessible via the file system, though customers should be aware that it must be in a format understandable by
the types in this package or behavior will be undefined.
AzureFileSystemProvider.newFileSystem(URI, Map)
will check for the following keys in the configuration map and expect the named
types. Any entries not listed here will be ignored. Note that AzureFileSystem
has public constants defined
for each of the keys for convenience.
AzureStorageSharedKeyCredential:
StorageSharedKeyCredential
AzureStorageSasTokenCredential:
AzureSasCredential
AzureStorageHttpLogDetailLevel:
HttpLogDetailLevel
AzureStorageMaxTries:
Integer
AzureStorageTryTimeout:
Integer
AzureStorageRetryDelayInMs:
Long
AzureStorageMaxRetryDelayInMs:
Long
AzureStorageRetryPolicyType:
RetryPolicyType
AzureStorageSecondaryHost:
String
AzureStorageSecondaryHost:
Integer
AzureStorageBlockSize:
Long
AzureStoragePutBlobThreshold:
Long
AzureStorageMaxConcurrencyPerRequest:
Integer
AzureStorageDownloadResumeRetries:
Integer
AzureStorageFileStores:
String
Either an account key or a sas token must be specified. If both are provided, the account key will be preferred. If a sas token is specified, the customer must take care that it has appropriate permissions to perform the actions demanded of the file system in a given workflow, including the initial connection check specified above. The same token will be applied to all operations.
An iterable of file stores must also be provided; each entry should simply be the name of a container. The first
container listed will be considered the default file store and the root directory of which will be the file system's
default directory. All other values listed are used to configure the underlying
BlobServiceClient
. Please refer to that type for more information on these values.
FileSystemProvider
Modifier and Type | Field and Description |
---|---|
static String |
CACHE_CONTROL
A helper for setting the HTTP properties when creating a directory.
|
static String |
CONTENT_DISPOSITION
A helper for setting the HTTP properties when creating a directory.
|
static String |
CONTENT_ENCODING
A helper for setting the HTTP properties when creating a directory.
|
static String |
CONTENT_LANGUAGE
A helper for setting the HTTP properties when creating a directory.
|
static String |
CONTENT_MD5
A helper for setting the HTTP properties when creating a directory.
|
static String |
CONTENT_TYPE
A helper for setting the HTTP properties when creating a directory.
|
Constructor and Description |
---|
AzureFileSystemProvider()
Creates an AzureFileSystemProvider.
|
Modifier and Type | Method and Description |
---|---|
void |
checkAccess(Path path,
AccessMode... accessModes)
Checks the existence, and optionally the accessibility, of a file.
|
void |
copy(Path source,
Path destination,
CopyOption... copyOptions)
Copies the resource at the source location to the destination.
|
void |
createDirectory(Path path,
FileAttribute<?>... fileAttributes)
Creates a new directory at the specified path.
|
void |
delete(Path path)
Deletes the specified resource.
|
<V extends FileAttributeView> |
getFileAttributeView(Path path,
Class<V> type,
LinkOption... linkOptions)
Returns a file attribute view of a given type.
|
FileStore |
getFileStore(Path path)
Unsupported.
|
FileSystem |
getFileSystem(URI uri)
Returns an existing FileSystem created by this provider.
|
Path |
getPath(URI uri)
Return a Path object by converting the given URI.
|
String |
getScheme()
Returns the URI scheme that identifies this provider:
"azb". |
boolean |
isHidden(Path path)
Always returns false as hidden files are not supported.
|
boolean |
isSameFile(Path path,
Path path1)
Unsupported.
|
void |
move(Path path,
Path path1,
CopyOption... copyOptions)
Unsupported.
|
SeekableByteChannel |
newByteChannel(Path path,
Set<? extends OpenOption> set,
FileAttribute<?>... fileAttributes)
Opens or creates a file, returning a seekable byte channel to access the file.
|
DirectoryStream<Path> |
newDirectoryStream(Path path,
DirectoryStream.Filter<? super Path> filter)
Returns an
AzureDirectoryStream for iterating over the contents of a directory. |
FileSystem |
newFileSystem(URI uri,
Map<String,?> config)
Constructs a new FileSystem object identified by a URI.
|
InputStream |
newInputStream(Path path,
OpenOption... options)
Opens an
InputStream to the given path. |
OutputStream |
newOutputStream(Path path,
OpenOption... options)
Opens an
OutputStream to the given path. |
<A extends BasicFileAttributes> |
readAttributes(Path path,
Class<A> type,
LinkOption... linkOptions)
Reads a file's attributes as a bulk operation.
|
Map<String,Object> |
readAttributes(Path path,
String attributes,
LinkOption... linkOptions)
Reads a set of file attributes as a bulk operation.
|
void |
setAttribute(Path path,
String attributes,
Object value,
LinkOption... linkOptions)
Sets the value of a file attribute.
|
createLink, createSymbolicLink, deleteIfExists, installedProviders, newAsynchronousFileChannel, newFileChannel, newFileSystem, readSymbolicLink
public static final String CONTENT_TYPE
public static final String CONTENT_DISPOSITION
public static final String CONTENT_LANGUAGE
public static final String CONTENT_ENCODING
public static final String CONTENT_MD5
public static final String CACHE_CONTROL
public AzureFileSystemProvider()
public String getScheme()
"azb".
getScheme
in class FileSystemProvider
"azb"
public FileSystem newFileSystem(URI uri, Map<String,?> config) throws IOException
The format of a URI
identifying a file system is "azb://?endpoint=<endpoint>"
.
Once closed, a file system with the same identifier may be reopened.
newFileSystem
in class FileSystemProvider
uri
- URI referenceconfig
- A map of provider specific properties to configure the file systemIllegalArgumentException
- If the pre-conditions for the uri parameter aren't met, or the env parameter
does not contain properties required by the provider, or a property value is invalid.IOException
- If an I/O error occurs.SecurityException
- neverFileSystemAlreadyExistsException
- If the file system has already been created.public FileSystem getFileSystem(URI uri)
The format of a URI
identifying an file system is "azb://?endpoint=<endpoint>"
.
Trying to retrieve a closed file system will throw a FileSystemNotFoundException
. Once closed, a
file system with the same identifier may be reopened.
getFileSystem
in class FileSystemProvider
uri
- URI referenceIllegalArgumentException
- If the pre-conditions for the uri parameter aren't metFileSystemNotFoundException
- If the file system already existsSecurityException
- neverpublic Path getPath(URI uri)
getPath
in class FileSystemProvider
uri
- The URI to convertIllegalArgumentException
- If the URI scheme does not identify this provider or other preconditions on the
uri parameter do not holdFileSystemNotFoundException
- if the file system identified by the query does not existSecurityException
- neverfor information on the URI format
public SeekableByteChannel newByteChannel(Path path, Set<? extends OpenOption> set, FileAttribute<?>... fileAttributes) throws IOException
This method is primarily offered to support some jdk convenience methods such as
Files.createFile(Path, FileAttribute[])
which requires opening a channel and closing it. A channel may
only be opened in read mode OR write mode. It may not be opened in read/write mode. Seeking is supported for
reads, but not for writes. Modifications to existing files is not permitted--only creating new files or
overwriting existing files.
This type is not threadsafe to prevent having to hold locks across network calls.
newByteChannel
in class FileSystemProvider
path
- the path of the file to openset
- options specifying how the file should be openedfileAttributes
- an optional list of file attributes to set atomically when creating the directoryUnsupportedOperationException
- Operation is not supported.IllegalArgumentException
- if the set contains an invalid combination of optionsFileAlreadyExistsException
- if a file of that name already exists and the CREATE_NEW option is specified
(optional specific exception)IOException
- If an I/O error occurs.SecurityException
- neverpublic InputStream newInputStream(Path path, OpenOption... options) throws IOException
InputStream
to the given path.
The stream will not attempt to read or buffer the entire file. However, when fetching data, it will always request the same size chunk of several MB to prevent network thrashing on small reads. Mark and reset are supported.
Only StandardOpenOption.READ
is supported. Any other option will throw.
newInputStream
in class FileSystemProvider
path
- the path to the file to openoptions
- options specifying how the file is openedIllegalArgumentException
- if an invalid combination of options is specifiedUnsupportedOperationException
- if an unsupported option is specifiedIOException
- If an I/O error occurs.SecurityException
- neverpublic OutputStream newOutputStream(Path path, OpenOption... options) throws IOException
OutputStream
to the given path. The resulting file will be stored as a block blob.
The only supported options are StandardOpenOption.CREATE
, StandardOpenOption.CREATE_NEW
,
StandardOpenOption.WRITE
, StandardOpenOption.TRUNCATE_EXISTING
. Any other options will throw an
UnsupportedOperationException
. WRITE
and TRUNCATE_EXISTING
must be specified or an
IllegalArgumentException
will be thrown. Hence, files cannot be updated, only overwritten completely.
This stream will not attempt to buffer the entire file, however some buffering will be done for potential
optimizations and to avoid network thrashing. Specifically, up to
AzureFileSystem.AZURE_STORAGE_PUT_BLOB_THRESHOLD
bytes will be buffered initially. If that threshold is
exceeded, the data will be broken into chunks and sent in blocks, and writes will be buffered into sizes of
AzureFileSystem.AZURE_STORAGE_UPLOAD_BLOCK_SIZE
. The maximum number of buffers of this size to be
allocated is defined by AzureFileSystem.AZURE_STORAGE_MAX_CONCURRENCY_PER_REQUEST
, which also configures
the level of parallelism with which we may write and thus may affect write speeds as well.
The data is only committed when the steam is closed. Hence data cannot be read from the destination until the stream is closed. When the close method returns, it is guaranteed that, barring any errors, the data is finalized and available for reading.
Writing happens asynchronously. Bytes passed for writing are stored until either the threshold or block size are met at which time they are sent to the service. When the write method returns, there is no guarantee about which phase of this process the data is in other than it has been accepted and will be written. Again, closing will guarantee that the data is written and available.
Flush is a no-op as regards data transfers, but it can be used to check the state of the stream for errors. This can be a useful tool because writing happens asynchronously, and therefore an error from a previous write may not otherwise be thrown unless the stream is flushed, closed, or written to again.
newOutputStream
in class FileSystemProvider
path
- the path to the file to open or createoptions
- options specifying how the file is openedIllegalArgumentException
- if an invalid combination of options is specifiedUnsupportedOperationException
- if an unsupported option is specifiedIOException
- If an I/O error occurs.SecurityException
- neverpublic DirectoryStream<Path> newDirectoryStream(Path path, DirectoryStream.Filter<? super Path> filter) throws IOException
AzureDirectoryStream
for iterating over the contents of a directory. The elements returned by
the directory stream's iterator are of type Path, each one representing an entry in the directory. The Path
objects are obtained as if by resolving the name of the directory entry against dir. The entries returned by the
iterator are filtered by the given filter.
When not using the try-with-resources construct, then directory stream's close method should be invoked after iteration is completed so as to free any resources held for the open directory.
Where the filter terminates due to an uncaught error or runtime exception then it is propagated to the hasNext or next method. Where an IOException is thrown, it results in the hasNext or next method throwing a DirectoryIteratorException with the IOException as the cause.
newDirectoryStream
in class FileSystemProvider
path
- the path to the directoryfilter
- the directory stream filterDirectoryStream
objectIllegalArgumentException
- If the path type is not an instance of AzurePath
.NotDirectoryException
- if the file could not otherwise be opened because it is not a directoryIOException
- If an I/O error occurs.SecurityException
- neverpublic void createDirectory(Path path, FileAttribute<?>... fileAttributes) throws IOException
The existence of a directory in the AzureFileSystem
is defined on two levels. Weak existence is
defined by the presence of a non-zero number of blobs prefixed with the directory's path. This concept is also
known as a virtual directory and enables the file system to work with containers that were pre-loaded
with data by another source but need to be accessed by this file system. Strong existence is defined as
the presence of an actual storage resource at the given path, which in the case of directories, is a zero-length
blob whose name is the directory path with a particular metadata field indicating the blob's status as a
directory. This is also known as a concrete directory. Directories created by this file system will
strongly exist. Operations targeting directories themselves as the object (e.g. setting properties) will target
marker blobs underlying concrete directories. Other operations (e.g. listing) will operate on the blob-name
prefix.
This method fulfills the nio contract of: "The check for the existence of the file and the creation of the directory if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the directory." More specifically, this method will atomically check for strong existence of another file or directory at the given path and fail if one is present. On the other hand, we only check for weak existence of the parent to determine if the given path is valid. Additionally, the action of checking whether the parent exists, is not atomic with the creation of the directory. Note that while it is possible that the parent may be deleted between when the parent is determined to exist and the creation of the child, the creation of the child will always ensure the existence of a virtual parent, so the child will never be left floating and unreachable. The different checks on parent and child is due to limitations in the Storage service API.
There may be some unintuitive behavior when working with directories in this file system, particularly virtual directories (usually those not created by this file system). A virtual directory will disappear as soon as all its children have been deleted. Furthermore, if a directory with the given path weakly exists at the time of calling this method, this method will still return success and create a concrete directory at the target location. In other words, it is possible to "double create" a directory if it first weakly exists and then is strongly created. This is both because it is impossible to atomically check if a virtual directory exists while creating a concrete directory and because such behavior will have minimal side effects--no files will be overwritten and the directory will still be available for writing as intended, though it may not be empty. This is not a complete list of such unintuitive behavior.
This method will attempt to extract standard HTTP content headers from the list of file attributes to set them
as blob headers. All other attributes will be set as blob metadata. The value of every attribute will be
converted to a String
with the exception of the Content-MD5 attribute which expects a byte[]
.
When extracting the content headers, the following strings will be used for comparison (constants for these
values can be found on this type):
Content-Type
Content-Disposition
Content-Language
Content-Encoding
Content-MD5
Cache-Control
createDirectory
in class FileSystemProvider
path
- the directory to createfileAttributes
- an optional list of file attributes to set atomically when creating the directoryIllegalArgumentException
- If the path type is not an instance of AzurePath
.UnsupportedOperationException
- if the array contains an attribute that cannot be set atomically when
creating the directoryFileAlreadyExistsException
- if a directory could not otherwise be created because a file of that name
already existsIOException
- If an I/O error occurs.SecurityException
- neverpublic void delete(Path path) throws IOException
This method is not atomic with respect to other file system operations. It is possible to delete a file in use by another process, and doing so will not immediately invalidate any channels open to that file--they will simply start to fail. Root directories cannot be deleted even when empty.
delete
in class FileSystemProvider
path
- the path to the file to deleteIllegalArgumentException
- If the path type is not an instance of AzurePath
.NoSuchFileException
- if the file does not existDirectoryNotEmptyException
- if the file is a directory and could not otherwise be deleted because the
directory is not emptyIOException
- If an I/O error occurs.SecurityException
- neverpublic void copy(Path source, Path destination, CopyOption... copyOptions) throws IOException
This method is not atomic with respect to other file system operations. More specifically, the checks necessary to validate the inputs and state of the file system are not atomic with the actual copying of data. If the copy is triggered, the copy itself is atomic and only a complete copy will ever be left at the destination.
In addition to those in the docs for FileSystemProvider.copy(Path, Path, CopyOption...)
, this method has
the following requirements for successful completion. StandardCopyOption.COPY_ATTRIBUTES
must be passed
as it is impossible not to copy blob properties; if this option is not passed, an
UnsupportedOperationException
will be thrown. Neither the source nor the destination can be a root
directory; if either is a root directory, an IllegalArgumentException
will be thrown. The parent
directory of the destination must at least weakly exist; if it does not, an IOException
will be thrown.
The only supported option other than StandardCopyOption.COPY_ATTRIBUTES
is
StandardCopyOption.REPLACE_EXISTING
; the presence of any other option will result in an
UnsupportedOperationException
.
This method supports both virtual and concrete directories as both the source and destination. Unlike when creating a directory, the existence of a virtual directory at the destination will cause this operation to fail. This is in order to prevent the possibility of overwriting a non-empty virtual directory with a file. Still, as mentioned above, this check is not atomic with the creation of the resultant directory.
copy
in class FileSystemProvider
source
- the path to the file to copydestination
- the path to the target filecopyOptions
- specifying how the copy should be doneUnsupportedOperationException
- if the array contains a copy option that is not supportedFileAlreadyExistsException
- if the target file exists but cannot be replaced because the REPLACE_EXISTING
option is not specifiedDirectoryNotEmptyException
- the REPLACE_EXISTING option is specified but the file cannot be replaced
because it is a non-empty directoryIOException
- If an I/O error occurs.IllegalArgumentException
- If the path type is not an instance of AzurePath
.SecurityException
- neverfor more information about directory existence.
public void move(Path path, Path path1, CopyOption... copyOptions) throws IOException
move
in class FileSystemProvider
path
- pathpath1
- pathcopyOptions
- optionsUnsupportedOperationException
- Operation is not supported.IOException
public boolean isSameFile(Path path, Path path1) throws IOException
isSameFile
in class FileSystemProvider
path
- pathpath1
- pathUnsupportedOperationException
- Operation is not supported.IOException
public boolean isHidden(Path path) throws IOException
isHidden
in class FileSystemProvider
path
- the pathIOException
- If an I/O error occurs.SecurityException
- neverpublic FileStore getFileStore(Path path) throws IOException
getFileStore
in class FileSystemProvider
path
- pathUnsupportedOperationException
- Operation is not supported.IOException
- If an I/O error occurs.SecurityException
- neverpublic void checkAccess(Path path, AccessMode... accessModes) throws IOException
This method may only be used to check the existence of a file. It is not possible to determine the permissions
granted to a given client, so if any mode argument is specified, an UnsupportedOperationException
will be
thrown.
checkAccess
in class FileSystemProvider
path
- the path to the file to checkaccessModes
- The access modes to check; may have zero elementsNoSuchFileException
- if a file does not existAccessDeniedException
- the requested access would be denied or the access cannot be
determined because the Java virtual machine has insufficient privileges or other reasonsIOException
- If an I/O error occurs.SecurityException
- neverpublic <V extends FileAttributeView> V getFileAttributeView(Path path, Class<V> type, LinkOption... linkOptions)
See AzureBasicFileAttributeView
and AzureBlobFileAttributeView
for more information.
Reading or setting attributes on a virtual directory is not supported and will throw an IOException
. See
AzureFileSystemProvider.createDirectory(Path, FileAttribute[])
for more information on virtual directories.
getFileAttributeView
in class FileSystemProvider
path
- the path to the filetype
- the Class object corresponding to the file attribute viewlinkOptions
- ignoredpublic <A extends BasicFileAttributes> A readAttributes(Path path, Class<A> type, LinkOption... linkOptions) throws IOException
See AzureBasicFileAttributes
and AzureBlobFileAttributes
for more information.
Reading attributes on a virtual directory is not supported and will throw an IOException
. See
AzureFileSystemProvider.createDirectory(Path, FileAttribute[])
for more information on virtual directories.
readAttributes
in class FileSystemProvider
path
- the path to the filetype
- the Class of the file attributes required to readlinkOptions
- ignoredUnsupportedOperationException
- if an attributes of the given type are not supportedIOException
- If an I/O error occurs.SecurityException
- neverpublic Map<String,Object> readAttributes(Path path, String attributes, LinkOption... linkOptions) throws IOException
See AzureBasicFileAttributes
and AzureBlobFileAttributes
for more information.
Reading attributes on a virtual directory is not supported and will throw an IOException
. See
AzureFileSystemProvider.createDirectory(Path, FileAttribute[])
for more information on virtual directories.
readAttributes
in class FileSystemProvider
path
- the path to the fileattributes
- the attributes to readlinkOptions
- ignoredUnsupportedOperationException
- if an attributes of the given type are not supportedIllegalArgumentException
- if no attributes are specified or an unrecognized attributes is specifiedIOException
- If an I/O error occurs.SecurityException
- neverpublic void setAttribute(Path path, String attributes, Object value, LinkOption... linkOptions) throws IOException
See AzureBlobFileAttributeView
for more information.
Setting attributes on a virtual directory is not supported and will throw an IOException
. See
AzureFileSystemProvider.createDirectory(Path, FileAttribute[])
for more information on virtual directories.
setAttribute
in class FileSystemProvider
path
- the path to the fileattributes
- the attribute to setvalue
- the attribute valuelinkOptions
- ignoredUnsupportedOperationException
- if an attribute view is not availableIllegalArgumentException
- if the attribute name is not specified, or is not recognized, or the attribute
value is of the correct type but has an inappropriate valueClassCastException
- If the attribute value is not of the expected type or is a collection containing
elements that are not of the expected typeIOException
- If an I/O error occurs.SecurityException
- neverCopyright © 2021 Microsoft Corporation. All rights reserved.