Package com.azure.storage.blob.nio

Class AzureFileSystemProvider

  • public final class AzureFileSystemProvider
extends FileSystemProvider
    The AzureFileSystemProvider is Azure Storage's implementation of the nio interface on top of Azure Blob Storage.

    Particular care should be taken when working with a remote storage service. This implementation makes no guarantees on behavior or state should other processes operate on the same data concurrently; file systems from this provider will assume they have exclusive access to their data and will behave without regard for potential of interfering applications. Moreover, remote file stores introduce higher latencies. Therefore, additional consideration should be given to managing concurrency: race conditions are more likely to manifest and network failures occur more frequently than disk failures. These and other such distributed application scenarios must be considered when working with this file system. While the AzureFileSystem will ensure it takes appropriate steps towards robustness and reliability, the application developer must design around these failure scenarios and have fallback and retry options available.

    The Azure Blob Storage service backing these APIs is not a true FileSystem, nor is it the goal of this implementation to force Azure Blob Storage to act like a full-fledged file system. Some APIs and scenarios will remain unsupported indefinitely until they may be sensibly implemented. Other APIs may experience lower performance than is expected because of the number of network requests needed to ensure correctness. The javadocs for each type and method should also be read carefully to understand what guarantees are made and how they may differ from the contract defined by FileSystemProvider.

    The scheme for this provider is "azb", and the format of the URI to identify an AzureFileSystem is "azb://?endpoint=<endpoing>". The endpoint of the Storage account is used to uniquely identify the filesystem.

    An AzureFileSystem is backed by an account. An AzureFileStore is backed by a container. Any number of containers may be specified as file stores upon creation of the file system. When a file system is created, it will try to retrieve the properties of each container to ensure connection to the account. If any of the containers does not exist, it will be created. Failure to access or create containers as necessary will result in an exception and failure to create the file system. Any data existing in the containers will be preserved and accessible via the file system, though customers should be aware that it must be in a format understandable by the types in this package or behavior will be undefined.

    newFileSystem(URI, Map) will check for the following keys in the configuration map and expect the named types. Any entries not listed here will be ignored. Note that AzureFileSystem has public constants defined for each of the keys for convenience.

    • AzureStorageSharedKeyCredential:StorageSharedKeyCredential
    • AzureStorageSasTokenCredential:AzureSasCredential
    • AzureStorageHttpLogDetailLevel:HttpLogDetailLevel
    • AzureStorageMaxTries:Integer
    • AzureStorageTryTimeout:Integer
    • AzureStorageRetryDelayInMs:Long
    • AzureStorageMaxRetryDelayInMs:Long
    • AzureStorageRetryPolicyType:RetryPolicyType
    • AzureStorageSecondaryHost:String
    • AzureStorageSecondaryHost:Integer
    • AzureStorageBlockSize:Long
    • AzureStoragePutBlobThreshold:Long
    • AzureStorageMaxConcurrencyPerRequest:Integer
    • AzureStorageDownloadResumeRetries:Integer
    • AzureStorageFileStores:String

    Either an account key or a sas token must be specified. If both are provided, the account key will be preferred. If a sas token is specified, the customer must take care that it has appropriate permissions to perform the actions demanded of the file system in a given workflow, including the initial connection check specified above. The same token will be applied to all operations.

    An iterable of file stores must also be provided; each entry should simply be the name of a container. The first container listed will be considered the default file store and the root directory of which will be the file system's default directory. All other values listed are used to configure the underlying BlobServiceClient. Please refer to that type for more information on these values.

    See Also:
    FileSystemProvider

    • Field Detail

      • CONTENT_TYPE

        public static final String CONTENT_TYPE
        A helper for setting the HTTP properties when creating a directory.
        See Also:
        Constant Field Values

      • CONTENT_DISPOSITION

        public static final String CONTENT_DISPOSITION
        A helper for setting the HTTP properties when creating a directory.
        See Also:
        Constant Field Values

      • CONTENT_LANGUAGE

        public static final String CONTENT_LANGUAGE
        A helper for setting the HTTP properties when creating a directory.
        See Also:
        Constant Field Values

      • CONTENT_ENCODING

        public static final String CONTENT_ENCODING
        A helper for setting the HTTP properties when creating a directory.
        See Also:
        Constant Field Values

      • CONTENT_MD5

        public static final String CONTENT_MD5
        A helper for setting the HTTP properties when creating a directory.
        See Also:
        Constant Field Values

      • CACHE_CONTROL

        public static final String CACHE_CONTROL
        A helper for setting the HTTP properties when creating a directory.
        See Also:
        Constant Field Values

    • Constructor Detail

      • AzureFileSystemProvider

        public AzureFileSystemProvider()
        Creates an AzureFileSystemProvider.

    • Method Detail

      • getScheme

        public String getScheme()
        Returns the URI scheme that identifies this provider: "azb".
        Specified by:
        getScheme in class FileSystemProvider
        Returns:
        "azb"

      • newFileSystem

        public FileSystem newFileSystem​(URI uri,
                                Map<String,​?> config)
                         throws IOException
        Constructs a new FileSystem object identified by a URI.

        The format of a URI identifying a file system is "azb://?endpoint=<endpoint>".

        Once closed, a file system with the same identifier may be reopened.

        Specified by:
        newFileSystem in class FileSystemProvider
        Parameters:
        uri - URI reference
        config - A map of provider specific properties to configure the file system
        Returns:
        a new file system.
        Throws:
        IllegalArgumentException - If the pre-conditions for the uri parameter aren't met, or the env parameter does not contain properties required by the provider, or a property value is invalid.
        IOException - If an I/O error occurs.
        SecurityException - never
        FileSystemAlreadyExistsException - If the file system has already been created.

      • newByteChannel

        public SeekableByteChannel newByteChannel​(Path path,
                                          Set<? extends OpenOption> set,
                                          FileAttribute<?>... fileAttributes)
                                   throws IOException
        Opens or creates a file, returning a seekable byte channel to access the file.

        This method is primarily offered to support some jdk convenience methods such as Files.createFile(Path, FileAttribute[]) which requires opening a channel and closing it. A channel may only be opened in read mode OR write mode. It may not be opened in read/write mode. Seeking is supported for reads, but not for writes. Modifications to existing files is not permitted--only creating new files or overwriting existing files.

        This type is not threadsafe to prevent having to hold locks across network calls.

        Specified by:
        newByteChannel in class FileSystemProvider
        Parameters:
        path - the path of the file to open
        set - options specifying how the file should be opened
        fileAttributes - an optional list of file attributes to set atomically when creating the directory
        Returns:
        a new seekable byte channel
        Throws:
        UnsupportedOperationException - Operation is not supported.
        IllegalArgumentException - if the set contains an invalid combination of options
        FileAlreadyExistsException - if a file of that name already exists and the CREATE_NEW option is specified (optional specific exception)
        IOException - If an I/O error occurs.
        SecurityException - never

      • newOutputStream

        public OutputStream newOutputStream​(Path path,
                                    OpenOption... options)
                             throws IOException
        Opens an OutputStream to the given path. The resulting file will be stored as a block blob.

        The only supported options are StandardOpenOption.CREATE, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING. Any other options will throw an UnsupportedOperationException. WRITE and TRUNCATE_EXISTING must be specified or an IllegalArgumentException will be thrown. Hence, files cannot be updated, only overwritten completely.

        This stream will not attempt to buffer the entire file, however some buffering will be done for potential optimizations and to avoid network thrashing. Specifically, up to AzureFileSystem.AZURE_STORAGE_PUT_BLOB_THRESHOLD bytes will be buffered initially. If that threshold is exceeded, the data will be broken into chunks and sent in blocks, and writes will be buffered into sizes of AzureFileSystem.AZURE_STORAGE_UPLOAD_BLOCK_SIZE. The maximum number of buffers of this size to be allocated is defined by AzureFileSystem.AZURE_STORAGE_MAX_CONCURRENCY_PER_REQUEST, which also configures the level of parallelism with which we may write and thus may affect write speeds as well.

        The data is only committed when the steam is closed. Hence data cannot be read from the destination until the stream is closed. When the close method returns, it is guaranteed that, barring any errors, the data is finalized and available for reading.

        Writing happens asynchronously. Bytes passed for writing are stored until either the threshold or block size are met at which time they are sent to the service. When the write method returns, there is no guarantee about which phase of this process the data is in other than it has been accepted and will be written. Again, closing will guarantee that the data is written and available.

        Flush is a no-op as regards data transfers, but it can be used to check the state of the stream for errors. This can be a useful tool because writing happens asynchronously, and therefore an error from a previous write may not otherwise be thrown unless the stream is flushed, closed, or written to again.

        Overrides:
        newOutputStream in class FileSystemProvider
        Parameters:
        path - the path to the file to open or create
        options - options specifying how the file is opened
        Returns:
        a new output stream
        Throws:
        IllegalArgumentException - if an invalid combination of options is specified
        UnsupportedOperationException - if an unsupported option is specified
        IOException - If an I/O error occurs.
        SecurityException - never

      • newDirectoryStream

        public DirectoryStream<Path> newDirectoryStream​(Path path,
                                                DirectoryStream.Filter<? super Path> filter)
                                         throws IOException
        Returns an AzureDirectoryStream for iterating over the contents of a directory. The elements returned by the directory stream's iterator are of type Path, each one representing an entry in the directory. The Path objects are obtained as if by resolving the name of the directory entry against dir. The entries returned by the iterator are filtered by the given filter.

        When not using the try-with-resources construct, then directory stream's close method should be invoked after iteration is completed so as to free any resources held for the open directory.

        Where the filter terminates due to an uncaught error or runtime exception then it is propagated to the hasNext or next method. Where an IOException is thrown, it results in the hasNext or next method throwing a DirectoryIteratorException with the IOException as the cause.

        Specified by:
        newDirectoryStream in class FileSystemProvider
        Parameters:
        path - the path to the directory
        filter - the directory stream filter
        Returns:
        a new and open DirectoryStream object
        Throws:
        IllegalArgumentException - If the path type is not an instance of AzurePath.
        NotDirectoryException - if the file could not otherwise be opened because it is not a directory
        IOException - If an I/O error occurs.
        SecurityException - never

      • createDirectory

        public void createDirectory​(Path path,
                            FileAttribute<?>... fileAttributes)
                     throws IOException
        Creates a new directory at the specified path.

        The existence of a directory in the AzureFileSystem is defined on two levels. Weak existence is defined by the presence of a non-zero number of blobs prefixed with the directory's path. This concept is also known as a virtual directory and enables the file system to work with containers that were pre-loaded with data by another source but need to be accessed by this file system. Strong existence is defined as the presence of an actual storage resource at the given path, which in the case of directories, is a zero-length blob whose name is the directory path with a particular metadata field indicating the blob's status as a directory. This is also known as a concrete directory. Directories created by this file system will strongly exist. Operations targeting directories themselves as the object (e.g. setting properties) will target marker blobs underlying concrete directories. Other operations (e.g. listing) will operate on the blob-name prefix.

        This method fulfills the nio contract of: "The check for the existence of the file and the creation of the directory if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the directory." More specifically, this method will atomically check for strong existence of another file or directory at the given path and fail if one is present. On the other hand, we only check for weak existence of the parent to determine if the given path is valid. Additionally, the action of checking whether the parent exists, is not atomic with the creation of the directory. Note that while it is possible that the parent may be deleted between when the parent is determined to exist and the creation of the child, the creation of the child will always ensure the existence of a virtual parent, so the child will never be left floating and unreachable. The different checks on parent and child is due to limitations in the Storage service API.

        There may be some unintuitive behavior when working with directories in this file system, particularly virtual directories (usually those not created by this file system). A virtual directory will disappear as soon as all its children have been deleted. Furthermore, if a directory with the given path weakly exists at the time of calling this method, this method will still return success and create a concrete directory at the target location. In other words, it is possible to "double create" a directory if it first weakly exists and then is strongly created. This is both because it is impossible to atomically check if a virtual directory exists while creating a concrete directory and because such behavior will have minimal side effects--no files will be overwritten and the directory will still be available for writing as intended, though it may not be empty. This is not a complete list of such unintuitive behavior.

        This method will attempt to extract standard HTTP content headers from the list of file attributes to set them as blob headers. All other attributes will be set as blob metadata. The value of every attribute will be converted to a String with the exception of the Content-MD5 attribute which expects a byte[]. When extracting the content headers, the following strings will be used for comparison (constants for these values can be found on this type):

        • Content-Type
        • Content-Disposition
        • Content-Language
        • Content-Encoding
        • Content-MD5
        • Cache-Control
        Note that these properties also have a particular semantic in that if one is specified, all are updated. In other words, if any of the above is set, all those that are not set will be cleared. See the Azure Docs for more information.
        Specified by:
        createDirectory in class FileSystemProvider
        Parameters:
        path - the directory to create
        fileAttributes - an optional list of file attributes to set atomically when creating the directory
        Throws:
        IllegalArgumentException - If the path type is not an instance of AzurePath.
        UnsupportedOperationException - if the array contains an attribute that cannot be set atomically when creating the directory
        FileAlreadyExistsException - if a directory could not otherwise be created because a file of that name already exists
        IOException - If an I/O error occurs.
        SecurityException - never

      • delete

        public void delete​(Path path)
            throws IOException
        Deletes the specified resource.

        This method is not atomic with respect to other file system operations. It is possible to delete a file in use by another process, and doing so will not immediately invalidate any channels open to that file--they will simply start to fail. Root directories cannot be deleted even when empty.

        Specified by:
        delete in class FileSystemProvider
        Parameters:
        path - the path to the file to delete
        Throws:
        IllegalArgumentException - If the path type is not an instance of AzurePath.
        NoSuchFileException - if the file does not exist
        DirectoryNotEmptyException - if the file is a directory and could not otherwise be deleted because the directory is not empty
        IOException - If an I/O error occurs.
        SecurityException - never

      • copy

        public void copy​(Path source,
                 Path destination,
                 CopyOption... copyOptions)
          throws IOException
        Copies the resource at the source location to the destination.

        This method is not atomic with respect to other file system operations. More specifically, the checks necessary to validate the inputs and state of the file system are not atomic with the actual copying of data. If the copy is triggered, the copy itself is atomic and only a complete copy will ever be left at the destination.

        In addition to those in the docs for FileSystemProvider.copy(Path, Path, CopyOption...), this method has the following requirements for successful completion. StandardCopyOption.COPY_ATTRIBUTES must be passed as it is impossible not to copy blob properties; if this option is not passed, an UnsupportedOperationException will be thrown. Neither the source nor the destination can be a root directory; if either is a root directory, an IllegalArgumentException will be thrown. The parent directory of the destination must at least weakly exist; if it does not, an IOException will be thrown. The only supported option other than StandardCopyOption.COPY_ATTRIBUTES is StandardCopyOption.REPLACE_EXISTING; the presence of any other option will result in an UnsupportedOperationException.

        This method supports both virtual and concrete directories as both the source and destination. Unlike when creating a directory, the existence of a virtual directory at the destination will cause this operation to fail. This is in order to prevent the possibility of overwriting a non-empty virtual directory with a file. Still, as mentioned above, this check is not atomic with the creation of the resultant directory.

        Specified by:
        copy in class FileSystemProvider
        Parameters:
        source - the path to the file to copy
        destination - the path to the target file
        copyOptions - specifying how the copy should be done
        Throws:
        UnsupportedOperationException - if the array contains a copy option that is not supported
        FileAlreadyExistsException - if the target file exists but cannot be replaced because the REPLACE_EXISTING option is not specified
        DirectoryNotEmptyException - the REPLACE_EXISTING option is specified but the file cannot be replaced because it is a non-empty directory
        IOException - If an I/O error occurs.
        IllegalArgumentException - If the path type is not an instance of AzurePath.
        SecurityException - never
        See Also:
        for more information about directory existence.

      • checkAccess

        public void checkAccess​(Path path,
                        AccessMode... accessModes)
                 throws IOException
        Checks the existence, and optionally the accessibility, of a file.

        This method may only be used to check the existence of a file. It is not possible to determine the permissions granted to a given client, so if any mode argument is specified, an UnsupportedOperationException will be thrown.

        Specified by:
        checkAccess in class FileSystemProvider
        Parameters:
        path - the path to the file to check
        accessModes - The access modes to check; may have zero elements
        Throws:
        NoSuchFileException - if a file does not exist
        AccessDeniedException - the requested access would be denied or the access cannot be determined because the Java virtual machine has insufficient privileges or other reasons
        IOException - If an I/O error occurs.
        SecurityException - never