.. role:: raw-html-m2r(raw) :format: html Azure Form Recognizer client library for Python =============================================== Azure Form Recognizer is a cloud service that uses machine learning to analyze text and structured data from your documents. It includes the following main features: * Layout - Extract content and structure (ex. words, selection marks, tables) from documents. * Document - Analyze key-value pairs in addition to general layout from documents. * Read - Read page information and detected languages from documents. * Prebuilt - Extract common field values from select document types (ex. receipts, invoices, business cards, ID documents, U.S. W-2 tax documents, among others) using prebuilt models. * Custom - Build custom models from your own data to extract tailored field values in addition to general layout from documents. `Source code `_ | `Package (PyPI) `_ | `API reference documentation `_ | `Product documentation `_ | `Samples `_ *Disclaimer* ---------------- *Azure SDK Python packages support for Python 2.7 ended 01 January 2022. For more information and questions, please refer to https://github.com/Azure/azure-sdk-for-python/issues/20691* Getting started --------------- Prerequisites ^^^^^^^^^^^^^ * Python 3.7 or later is required to use this package. * You must have an `Azure subscription `_ and a `Cognitive Services or Form Recognizer resource `_ to use this package. Install the package ^^^^^^^^^^^^^^^^^^^ Install the Azure Form Recognizer client library for Python with `pip `_\ : .. code-block:: bash pip install azure-ai-formrecognizer .. Note: This version of the client library defaults to the ``2022-08-31`` version of the service. This table shows the relationship between SDK versions and supported API versions of the service: .. list-table:: :header-rows: 1 * - SDK version - Supported API version of service * - 3.2.0 - Latest GA release - 2.0, 2.1, 2022-08-31 (default) * - 3.1.X - 2.0, 2.1 (default) * - 3.0.0 - 2.0 .. Note: Starting with version ``3.2.X``\ , a new set of clients were introduced to leverage the newest features of the Form Recognizer service. Please see the `Migration Guide `_ for detailed instructions on how to update application code from client library version ``3.1.X`` or lower to the latest version. Additionally, see the `Changelog `_ for more detailed information. The below table describes the relationship of each client and its supported API version(s): .. list-table:: :header-rows: 1 * - API version - Supported clients * - 2022-08-31 - DocumentAnalysisClient and DocumentModelAdministrationClient * - 2.1 - FormRecognizerClient and FormTrainingClient * - 2.0 - FormRecognizerClient and FormTrainingClient Create a Cognitive Services or Form Recognizer resource ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Form Recognizer supports both `multi-service and single-service access `_. Create a Cognitive Services resource if you plan to access multiple cognitive services under a single endpoint/key. For Form Recognizer access only, create a Form Recognizer resource. Please note that you will need a single-service resource if you intend to use `Azure Active Directory authentication <#create-the-client-with-an-azure-active-directory-credential>`_. You can create either resource using: * Option 1: `Azure Portal `_. * Option 2: `Azure CLI `_. Below is an example of how you can create a Form Recognizer resource using the CLI: .. code-block:: PowerShell # Create a new resource group to hold the form recognizer resource # if using an existing resource group, skip this step az group create --name --location .. code-block:: PowerShell # Create form recognizer az cognitiveservices account create \ --name \ --resource-group \ --kind FormRecognizer \ --sku \ --location \ --yes For more information about creating the resource or how to get the location and sku information see `here `_. Authenticate the client ^^^^^^^^^^^^^^^^^^^^^^^ In order to interact with the Form Recognizer service, you will need to create an instance of a client. An **endpoint** and **credential** are necessary to instantiate the client object. Get the endpoint ~~~~~~~~~~~~~~~~ You can find the endpoint for your Form Recognizer resource using the `Azure Portal `_ or `Azure CLI `_\ : .. code-block:: bash # Get the endpoint for the form recognizer resource az cognitiveservices account show --name "resource-name" --resource-group "resource-group-name" --query "properties.endpoint" Either a regional endpoint or a custom subdomain can be used for authentication. They are formatted as follows: .. code-block:: Regional endpoint: https://.api.cognitive.microsoft.com/ Custom subdomain: https://.cognitiveservices.azure.com/ A regional endpoint is the same for every resource in a region. A complete list of supported regional endpoints can be consulted `here `_. Please note that regional endpoints do not support AAD authentication. A custom subdomain, on the other hand, is a name that is unique to the Form Recognizer resource. They can only be used by `single-service resources `_. Get the API key ~~~~~~~~~~~~~~~ The API key can be found in the `Azure Portal `_ or by running the following Azure CLI command: .. code-block:: bash az cognitiveservices account keys list --name "" --resource-group "" Create the client with AzureKeyCredential ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To use an `API key `_ as the ``credential`` parameter, pass the key as a string into an instance of `AzureKeyCredential `_. .. code-block:: python from azure.core.credentials import AzureKeyCredential from azure.ai.formrecognizer import DocumentAnalysisClient endpoint = "https://.cognitiveservices.azure.com/" credential = AzureKeyCredential("") document_analysis_client = DocumentAnalysisClient(endpoint, credential) Create the client with an Azure Active Directory credential ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``AzureKeyCredential`` authentication is used in the examples in this getting started guide, but you can also authenticate with Azure Active Directory using the `azure-identity `_ library. Note that regional endpoints do not support AAD authentication. Create a `custom subdomain `_ name for your resource in order to use this type of authentication. To use the `DefaultAzureCredential `_ type shown below, or other credential types provided with the Azure SDK, please install the ``azure-identity`` package: ``pip install azure-identity`` You will also need to `register a new AAD application and grant access `_ to Form Recognizer by assigning the ``"Cognitive Services User"`` role to your service principal. Once completed, set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables: ``AZURE_CLIENT_ID``\ , ``AZURE_TENANT_ID``\ , ``AZURE_CLIENT_SECRET``. .. code-block:: python from azure.identity import DefaultAzureCredential from azure.ai.formrecognizer import DocumentAnalysisClient credential = DefaultAzureCredential() document_analysis_client = DocumentAnalysisClient( endpoint="https://.cognitiveservices.azure.com/", credential=credential ) Key concepts ------------ DocumentAnalysisClient ^^^^^^^^^^^^^^^^^^^^^^ ``DocumentAnalysisClient`` provides operations for analyzing input documents using prebuilt and custom models through the ``begin_analyze_document`` and ``begin_analyze_document_from_url`` APIs. Use the ``model_id`` parameter to select the type of model for analysis. See a full list of supported models `here `_. Sample code snippets are provided to illustrate using a DocumentAnalysisClient :raw-html-m2r:`here`. More information about analyzing documents, including supported features, locales, and document types can be found in the `service documentation `_. DocumentModelAdministrationClient ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``DocumentModelAdministrationClient`` provides operations for: * Building custom models to analyze specific fields you specify by labeling your custom documents. A ``DocumentModelDetails`` is returned indicating the document type(s) the model can analyze, as well as the estimated confidence for each field. See the `service documentation `_ for a more detailed explanation. * Creating a composed model from a collection of existing models. * Managing models created in your account. * Listing operations or getting a specific model operation created within the last 24 hours. * Copying a custom model from one Form Recognizer resource to another. Please note that models can also be built using a graphical user interface such as `Form Recognizer Studio `_. Sample code snippets are provided to illustrate using a DocumentModelAdministrationClient :raw-html-m2r:`here`. Long-running operations ^^^^^^^^^^^^^^^^^^^^^^^ Long-running operations are operations which consist of an initial request sent to the service to start an operation, followed by polling the service at intervals to determine whether the operation has completed or failed, and if it has succeeded, to get the result. Methods that analyze documents, build models, or copy/compose models are modeled as long-running operations. The client exposes a ``begin_`` method that returns an ``LROPoller`` or ``AsyncLROPoller``. Callers should wait for the operation to complete by calling ``result()`` on the poller object returned from the ``begin_`` method. Sample code snippets are provided to illustrate using long-running operations :raw-html-m2r:`below`. Examples -------- The following section provides several code snippets covering some of the most common Form Recognizer tasks, including: * :raw-html-m2r:`Extract Layout` * :raw-html-m2r:`Using the General Document Model` * :raw-html-m2r:`Using Prebuilt Models` * :raw-html-m2r:`Build a Custom Model` * :raw-html-m2r:`Analyze Documents Using a Custom Model` * :raw-html-m2r:`Manage Your Models` Extract Layout ^^^^^^^^^^^^^^ Extract text, selection marks, text styles, and table structures, along with their bounding region coordinates, from documents. .. code-block:: python from azure.ai.formrecognizer import DocumentAnalysisClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.cognitiveservices.azure.com/" credential = AzureKeyCredential("") document_analysis_client = DocumentAnalysisClient(endpoint, credential) with open("", "rb") as fd: document = fd.read() poller = document_analysis_client.begin_analyze_document("prebuilt-layout", document) result = poller.result() for page in result.pages: print("----Analyzing layout from page #{}----".format(page.page_number)) print( "Page has width: {} and height: {}, measured with unit: {}".format( page.width, page.height, page.unit ) ) for line_idx, line in enumerate(page.lines): print( "...Line # {} has content '{}' within bounding polygon '{}'".format( line_idx, line.content, line.polygon, ) ) for word in page.words: print( "...Word '{}' has a confidence of {}".format( word.content, word.confidence ) ) for selection_mark in page.selection_marks: print( "...Selection mark is '{}' within bounding polygon '{}' and has a confidence of {}".format( selection_mark.state, selection_mark.polygon, selection_mark.confidence, ) ) for table_idx, table in enumerate(result.tables): print( "Table # {} has {} rows and {} columns".format( table_idx, table.row_count, table.column_count ) ) for region in table.bounding_regions: print( "Table # {} location on page: {} is {}".format( table_idx, region.page_number, region.polygon ) ) for cell in table.cells: print( "...Cell[{}][{}] has content '{}'".format( cell.row_index, cell.column_index, cell.content, ) ) Using the General Document Model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Analyze key-value pairs, tables, styles, and selection marks from documents using the general document model provided by the Form Recognizer service. Select the General Document Model by passing ``model_id="prebuilt-document"`` into the ``begin_analyze_document`` method: .. code-block:: python from azure.ai.formrecognizer import DocumentAnalysisClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.cognitiveservices.azure.com/" credential = AzureKeyCredential("") document_analysis_client = DocumentAnalysisClient(endpoint, credential) with open("", "rb") as fd: document = fd.read() poller = document_analysis_client.begin_analyze_document("prebuilt-document", document) result = poller.result() print("----Key-value pairs found in document----") for kv_pair in result.key_value_pairs: if kv_pair.key: print( "Key '{}' found within '{}' bounding regions".format( kv_pair.key.content, kv_pair.key.bounding_regions, ) ) if kv_pair.value: print( "Value '{}' found within '{}' bounding regions\n".format( kv_pair.value.content, kv_pair.value.bounding_regions, ) ) print("----Tables found in document----") for table_idx, table in enumerate(result.tables): print( "Table # {} has {} rows and {} columns".format( table_idx, table.row_count, table.column_count ) ) for region in table.bounding_regions: print( "Table # {} location on page: {} is {}".format( table_idx, region.page_number, region.polygon, ) ) print("----Styles found in document----") for style in result.styles: if style.is_handwritten: print("Document contains handwritten content: ") print(",".join([result.content[span.offset:span.offset + span.length] for span in style.spans])) for page in result.pages: print("----Analyzing document from page #{}----".format(page.page_number)) print( "Page has width: {} and height: {}, measured with unit: {}".format( page.width, page.height, page.unit ) ) for line_idx, line in enumerate(page.lines): words = line.get_words() print( "...Line # {} has {} words and text '{}' within bounding polygon '{}'".format( line_idx, len(words), line.content, line.polygon, ) ) for word in words: print( "......Word '{}' has a confidence of {}".format( word.content, word.confidence ) ) for selection_mark in page.selection_marks: print( "...Selection mark is '{}' within bounding polygon '{}' and has a confidence of {}".format( selection_mark.state, selection_mark.polygon, selection_mark.confidence, ) ) * Read more about the features provided by the ``prebuilt-document`` model `here `_. Using Prebuilt Models ^^^^^^^^^^^^^^^^^^^^^ Extract fields from select document types such as receipts, invoices, business cards, identity documents, and U.S. W-2 tax documents using prebuilt models provided by the Form Recognizer service. For example, to analyze fields from a sales receipt, use the prebuilt receipt model provided by passing ``model_id="prebuilt-receipt"`` into the ``begin_analyze_document`` method: .. code-block:: python from azure.ai.formrecognizer import DocumentAnalysisClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.cognitiveservices.azure.com/" credential = AzureKeyCredential("") document_analysis_client = DocumentAnalysisClient(endpoint, credential) with open("", "rb") as fd: receipt = fd.read() poller = document_analysis_client.begin_analyze_document("prebuilt-receipt", receipt) result = poller.result() for receipt in result.documents: for name, field in receipt.fields.items(): if name == "Items": print("Receipt Items:") for idx, item in enumerate(field.value): print("...Item #{}".format(idx+1)) for item_field_name, item_field in item.value.items(): print("......{}: {} has confidence {}".format( item_field_name, item_field.value, item_field.confidence)) else: print("{}: {} has confidence {}".format(name, field.value, field.confidence)) You are not limited to receipts! There are a few prebuilt models to choose from, each of which has its own set of supported fields. See other supported prebuilt models `here `_. Build a Custom Model ^^^^^^^^^^^^^^^^^^^^ Build a custom model on your own document type. The resulting model can be used to analyze values from the types of documents it was trained on. Provide a container SAS URL to your Azure Storage Blob container where you're storing the training documents. More details on setting up a container and required file structure can be found in the `service documentation `_. .. code-block:: python from azure.ai.formrecognizer import DocumentModelAdministrationClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.cognitiveservices.azure.com/" credential = AzureKeyCredential("") document_model_admin_client = DocumentModelAdministrationClient(endpoint, credential) container_sas_url = "" # training documents uploaded to blob storage poller = document_model_admin_client.begin_build_document_model( # For more information about build_mode, see: https://aka.ms/azsdk/formrecognizer/buildmode build_mode="template", blob_container_url=container_sas_url, model_id="my-first-model" ) model = poller.result() print("Model ID: {}".format(model.model_id)) print("Description: {}".format(model.description)) print("Model created on: {}\n".format(model.created_on)) print("Doc types the model can recognize:") for name, doc_type in model.doc_types.items(): print("\nDoc Type: '{}' which has the following fields:".format(name)) for field_name, confidence in doc_type.field_confidence.items(): print("Field: '{}' has confidence score {}".format(field_name, confidence)) Analyze Documents Using a Custom Model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Analyze document fields, tables, selection marks, and more. These models are trained with your own data, so they're tailored to your documents. For best results, you should only analyze documents of the same document type that the custom model was built with. .. code-block:: python from azure.ai.formrecognizer import DocumentAnalysisClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.cognitiveservices.azure.com/" credential = AzureKeyCredential("") document_analysis_client = DocumentAnalysisClient(endpoint, credential) model_id = "" with open("", "rb") as fd: document = fd.read() poller = document_analysis_client.begin_analyze_document(model_id=model_id, document=document) result = poller.result() for analyzed_document in result.documents: print("Document was analyzed by model with ID {}".format(result.model_id)) print("Document has confidence {}".format(analyzed_document.confidence)) for name, field in analyzed_document.fields.items(): print("Field '{}' has value '{}' with confidence of {}".format(name, field.value, field.confidence)) # iterate over lines, words, and selection marks on each page of the document for page in result.pages: print("\nLines found on page {}".format(page.page_number)) for line in page.lines: print("...Line '{}'".format(line.content)) print("\nWords found on page {}".format(page.page_number)) for word in page.words: print( "...Word '{}' has a confidence of {}".format( word.content, word.confidence ) ) print("\nSelection marks found on page {}".format(page.page_number)) for selection_mark in page.selection_marks: print( "...Selection mark is '{}' and has a confidence of {}".format( selection_mark.state, selection_mark.confidence ) ) # iterate over tables in document for i, table in enumerate(result.tables): print("\nTable {} can be found on page:".format(i + 1)) for region in table.bounding_regions: print("...{}".format(region.page_number)) for cell in table.cells: print( "...Cell[{}][{}] has content '{}'".format( cell.row_index, cell.column_index, cell.content ) ) Alternatively, a document URL can also be used to analyze documents using the ``begin_analyze_document_from_url`` method. .. code-block:: python document_url = "" poller = document_analysis_client.begin_analyze_document_from_url(model_id=model_id, document_url=document_url) result = poller.result() Manage Your Models ^^^^^^^^^^^^^^^^^^ Manage the custom models attached to your account. .. code-block:: python from azure.ai.formrecognizer import DocumentModelAdministrationClient from azure.core.credentials import AzureKeyCredential from azure.core.exceptions import ResourceNotFoundError endpoint = "https://.cognitiveservices.azure.com/" credential = AzureKeyCredential("") document_model_admin_client = DocumentModelAdministrationClient(endpoint, credential) account_details = document_model_admin_client.get_resource_details() print("Our account has {} custom models, and we can have at most {} custom models".format( account_details.custom_document_models.count, account_details.custom_document_models.limit )) # Here we get a paged list of all of our models models = document_model_admin_client.list_document_models() print("We have models with the following ids: {}".format( ", ".join([m.model_id for m in models]) )) # Replace with the custom model ID from the "Build a model" sample model_id = "" custom_model = document_model_admin_client.get_document_model(model_id=model_id) print("Model ID: {}".format(custom_model.model_id)) print("Description: {}".format(custom_model.description)) print("Model created on: {}\n".format(custom_model.created_on)) # Finally, we will delete this model by ID document_model_admin_client.delete_document_model(model_id=custom_model.model_id) try: document_model_admin_client.get_document_model(model_id=custom_model.model_id) except ResourceNotFoundError: print("Successfully deleted model with id {}".format(custom_model.model_id)) Troubleshooting --------------- General ^^^^^^^ Form Recognizer client library will raise exceptions defined in `Azure Core `_. Error codes and messages raised by the Form Recognizer service can be found in the `service documentation `_. Logging ^^^^^^^ This library uses the standard `logging `_ library for logging. Basic information about HTTP sessions (URLs, headers, etc.) is logged at ``INFO`` level. Detailed ``DEBUG`` level logging, including request/response bodies and **unredacted** headers, can be enabled on the client or per-operation with the ``logging_enable`` keyword argument. See full SDK logging documentation with examples `here `_. Optional Configuration ^^^^^^^^^^^^^^^^^^^^^^ Optional keyword arguments can be passed in at the client and per-operation level. The azure-core `reference documentation `_ describes available configurations for retries, logging, transport protocols, and more. Next steps ---------- More sample code ^^^^^^^^^^^^^^^^ See the `Sample README `_ for several code snippets illustrating common patterns used in the Form Recognizer Python API. Additional documentation ^^^^^^^^^^^^^^^^^^^^^^^^ For more extensive documentation on Azure Cognitive Services Form Recognizer, see the `Form Recognizer documentation `_ on docs.microsoft.com. Contributing ------------ This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit `cla.microsoft.com `_. When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. This project has adopted the `Microsoft Open Source Code of Conduct `_. For more information see the `Code of Conduct FAQ `_ or contact `opencode@microsoft.com `_ with any additional questions or comments. .. raw:: html Indices and tables ------------------ * :ref:`genindex` * :ref:`modindex` * :ref:`search` .. toctree:: :maxdepth: 5 :glob: :caption: Developer Documentation azure.ai.formrecognizer.rst