.. role:: raw-html-m2r(raw) :format: html Azure Form Recognizer client library for Python =============================================== Azure Cognitive Services Form Recognizer is a cloud service that uses machine learning to recognize text and table data from form documents. It includes the following main functionalities: * Custom models - Recognize field values and table data from forms. These models are trained with your own data, so they're tailored to your forms. You can then take these custom models and recognize forms. You can also manage the custom models you've created and see how close you are to the limit of custom models your account can hold. * Content API - Recognize text and table structures, along with their bounding box coordinates, from documents. Corresponds to the REST service's Layout API. * Prebuilt receipt model - Recognize data from USA sales receipts using a prebuilt model. `Source code `_ | `Package (PyPI) `_ | `API reference documentation `_\ | `Product documentation `_ | `Samples `_ Getting started --------------- Prerequisites ^^^^^^^^^^^^^ * Python 2.7, or 3.5 or later is required to use this package. * You must have an `Azure subscription `_ and a `Cognitive Services or Form Recognizer resource `_ to use this package. Install the package ^^^^^^^^^^^^^^^^^^^ Install the Azure Form Recognizer client library for Python with `pip `_\ : .. code-block:: bash pip install azure-ai-formrecognizer .. Note: This version of the client library supports the v2.0-preview version of the Form Recognizer service Create a Form Recognizer resource ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Form Recognizer supports both `multi-service and single-service access `_. Create a Cognitive Services resource if you plan to access multiple cognitive services under a single endpoint/key. For Form Recognizer access only, create a Form Recognizer resource. You can create the resource using **Option 1:** `Azure Portal `_ **Option 2:** `Azure CLI `_. Below is an example of how you can create a Form Recognizer resource using the CLI: .. code-block:: bash # Create a new resource group to hold the form recognizer resource - # if using an existing resource group, skip this step az group create --name my-resource-group --location westus2 .. code-block:: bash # Create form recognizer az cognitiveservices account create \ --name form-recognizer-resource \ --resource-group my-resource-group \ --kind FormRecognizer \ --sku F0 \ --location westus2 \ --yes Authenticate the client ^^^^^^^^^^^^^^^^^^^^^^^ Looking up the endpoint ~~~~~~~~~~~~~~~~~~~~~~~ You can find the endpoint for your form recognizer resource using the `Azure Portal `_ or `Azure CLI `_\ : .. code-block:: bash # Get the endpoint for the form recognizer resource az cognitiveservices account show --name "resource-name" --resource-group "resource-group-name" --query "endpoint" Types of credentials ~~~~~~~~~~~~~~~~~~~~ The ``credential`` parameter may be provided as a `AzureKeyCredential `_ from `azure.core `_\ , or as a credential type from Azure Active Directory. See the full details regarding `authentication `_ of cognitive services. #. To use an `API key `_\ , pass the key as a string into an instance of ``AzureKeyCredential("")``. The API key can be found in the Azure Portal or by running the following Azure CLI command: ``az cognitiveservices account keys list --name "resource-name" --resource-group "resource-group-name"`` Use the key as the credential parameter to authenticate the client: .. code-block:: python from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) #. To use an `Azure Active Directory (AAD) token credential `_\ , provide an instance of the desired credential type obtained from the `azure-identity `_ library. Note that regional endpoints do not support AAD authentication. Create a `custom subdomain `_ name for your resource in order to use this type of authentication. Authentication with AAD requires some initial setup: * `Install azure-identity `_ * `Register a new AAD application `_ * `Grant access `_ to Form Recognizer by assigning the ``"Cognitive Services User"`` role to your service principal. After setup, you can choose which type of `credential `_ from azure.identity to use. As an example, `DefaultAzureCredential `_ can be used to authenticate the client: Set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET Use the returned token credential to authenticate the client: .. code-block:: python from azure.identity import DefaultAzureCredential from azure.ai.formrecognizer import FormRecognizerClient token_credential = DefaultAzureCredential() form_recognizer_client = FormRecognizerClient( endpoint="https://.cognitiveservices.azure.com/", credential=token_credential ) Key concepts ------------ FormRecognizerClient ^^^^^^^^^^^^^^^^^^^^ ``FormRecognizerClient`` provides operations for: * Recognizing form fields and content using custom models trained to recognize your custom forms. These values are returned in a collection of ``RecognizedForm`` objects. * Recognizing common fields from US receipts, using a pre-trained receipt model on the Form Recognizer service. These fields and meta-data are returned in a collection of ``RecognizedForm`` objects. * Recognizing form content, including tables, lines and words, without the need to train a model. Form content is returned in a collection of ``FormPage`` objects. Sample code snippets are provided to illustrate using a FormRecognizerClient :raw-html-m2r:`here`. FormTrainingClient ^^^^^^^^^^^^^^^^^^ ``FormTrainingClient`` provides operations for: * Training custom models to recognize all fields and values found in your custom forms. A ``CustomFormModel`` is returned indicating the form types the model will recognize, and the fields it will extract for each form type. See the `service's documents `_ for a more detailed explanation. * Training custom models to recognize specific fields and values you specify by labeling your custom forms. A ``CustomFormModel`` is returned indicating the fields the model will extract, as well as the estimated accuracy for each field. See the `service's documents `_ for a more detailed explanation. * Managing models created in your account. * Copying a custom model from one Form Recognizer resource to another. Please note that models can also be trained using a graphical user interface such as the `Form Recognizer Labeling Tool `_. Sample code snippets are provided to illustrate using a FormTrainingClient :raw-html-m2r:`here`. Long-Running Operations ^^^^^^^^^^^^^^^^^^^^^^^ Long-running operations are operations which consist of an initial request sent to the service to start an operation, followed by polling the service at intervals to determine whether the operation has completed or failed, and if it has succeeded, to get the result. Methods that train models, recognize values from forms, or copy models are modeled as long-running operations. The client exposes a ``begin_`` method that returns an ``LROPoller`` or ``AsyncLROPoller``. Callers should wait for the operation to complete by calling ``result()`` on the operation returned from the ``begin_`` method. Sample code snippets are provided to illustrate using long-running operations :raw-html-m2r:`below`. Examples -------- The following section provides several code snippets covering some of the most common Form Recognizer tasks, including: * :raw-html-m2r:`Recognize Forms Using a Custom Model` * :raw-html-m2r:`Recognize Content` * :raw-html-m2r:`Recognize Receipts` * :raw-html-m2r:`Train a Model` * :raw-html-m2r:`Manage Your Models` Recognize Forms Using a Custom Model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Recognize name/value pairs and table data from forms. These models are trained with your own data, so they're tailored to your forms. You should only recognize forms of the same form type that the custom model was trained on. .. code-block:: python from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) model_id = "" # Make sure the form type is one of the types of forms your custom model can recognize with open("", "rb") as fd: form = fd.read() poller = form_recognizer_client.begin_recognize_custom_forms(model_id=model_id, form=form) result = poller.result() for recognized_form in result: print("Form type ID: {}".format(recognized_form.form_type)) for label, field in recognized_form.fields.items(): print("Field '{}' has value '{}' with a confidence score of {}".format( label, field.value, field.confidence )) Alternatively, a form url can also be used to recognize custom forms using the ``begin_recognize_custom_forms_from_url`` method. The ``_from_url`` methods exist for all the recognize methods. .. code-block:: form_url_jpg = "" poller = form_recognizer_client.begin_recognize_custom_forms_from_url(model_id=model_id, form_url=form_url) result = poller.result() Recognize Content ^^^^^^^^^^^^^^^^^ Recognize text and table structures, along with their bounding box coordinates, from documents. .. code-block:: python from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) with open("", "rb") as fd: form = fd.read() poller = form_recognizer_client.begin_recognize_content(form) page = poller.result() table = page[0].tables[0] # page 1, table 1 for cell in table.cells: print(cell.text) print(cell.bounding_box) print(cell.confidence) Recognize Receipts ^^^^^^^^^^^^^^^^^^ Recognize data from USA sales receipts using a prebuilt model. `Here `_ are the fields the service returns for a recognized receipt. .. code-block:: python from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) with open("", "rb") as fd: receipt = fd.read() poller = form_recognizer_client.begin_recognize_receipts(receipt) result = poller.result() for receipt in result: for name, field in receipt.fields.items(): if name == "Items": print("Receipt Items:") for idx, items in enumerate(field.value): print("...Item #{}".format(idx)) for item_name, item in items.value.items(): print("......{}: {} has confidence {}".format(item_name, item.value, item.confidence)) else: print("{}: {} has confidence {}".format(name, field.value, field.confidence)) Train a model ^^^^^^^^^^^^^ Train a machine-learned model on your own form type. The resulting model will be able to recognize values from the types of forms it was trained on. Provide a container SAS url to your Azure Storage Blob container where you're storing the training documents. If training files are within a subfolder in the container, use the `prefix `_ keyword argument to specify under which folder to train. More details on setting up a container and required file structure can be found in the `service quickstart documentation `_. .. code-block:: python from azure.ai.formrecognizer import FormTrainingClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_training_client = FormTrainingClient(endpoint, credential) container_sas_url = "" # training documents uploaded to blob storage poller = form_training_client.begin_training(container_sas_url, use_training_labels=False) model = poller.result() # Custom model information print("Model ID: {}".format(model.model_id)) print("Status: {}".format(model.status)) print("Training started on: {}".format(model.training_started_on)) print("Training completed on: {}".format(model.training_completed_on)) print("Recognized fields:") # looping through the submodels, which contains the fields they were trained on for submodel in model.submodels: print("The submodel with form type '{}' has recognized the following fields: {}".format( submodel.form_type, ", ".join([label for label in submodel.fields]) )) # Training result information for doc in model.training_documents: print("Document name: {}".format(doc.document_name)) print("Document status: {}".format(doc.status)) print("Document page count: {}".format(doc.page_count)) print("Document errors: {}".format(doc.errors)) Manage Your Models ^^^^^^^^^^^^^^^^^^ Manage the custom models attached to your account. .. code-block:: python from azure.ai.formrecognizer import FormTrainingClient from azure.core.credentials import AzureKeyCredential from azure.core.exceptions import ResourceNotFoundError endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_training_client = FormTrainingClient(endpoint, credential) account_properties = form_training_client.get_account_properties() print("Our account has {} custom models, and we can have at most {} custom models".format( account_properties.custom_model_count, account_properties.custom_model_limit )) # Here we get a paged list of all of our custom models custom_models = form_training_client.list_custom_models() print("We have models with the following ids: {}".format( ", ".join([m.model_id for m in custom_models]) )) # Now we get the custom model from the "Train a model" sample model_id = "" custom_model = form_training_client.get_custom_model(model_id=model_id) print("Model ID: {}".format(custom_model.model_id)) print("Status: {}".format(custom_model.status)) print("Training started on: {}".format(custom_model.training_started_on)) print("Training completed on: {}".format(custom_model.training_completed_on)) # Finally, we will delete this model by ID form_training_client.delete_model(model_id=custom_model.model_id) try: form_training_client.get_custom_model(model_id=custom_model.model_id) except ResourceNotFoundError: print("Successfully deleted model with id {}".format(custom_model.model_id)) Async APIs ---------- This library also includes a complete async API supported on Python 3.5+. To use it, you must first install an async transport, such as `aiohttp `_. See `azure-core documentation `_ for more information. Optional Configuration ---------------------- Optional keyword arguments can be passed in at the client and per-operation level. The azure-core `reference documentation `_ describes available configurations for retries, logging, transport protocols, and more. Troubleshooting --------------- General ^^^^^^^ Form Recognizer client library will raise exceptions defined in `Azure Core `_. Logging ^^^^^^^ This library uses the standard `logging `_ library for logging. Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO level. Detailed DEBUG level logging, including request/response bodies and unredacted headers, can be enabled on a client with the ``logging_enable`` keyword argument: .. code-block:: python import sys import logging from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential # Create a logger for the 'azure' SDK logger = logging.getLogger('azure') logger.setLevel(logging.DEBUG) # Configure a console output handler = logging.StreamHandler(stream=sys.stdout) logger.addHandler(handler) endpoint = "https://.cognitiveservices.azure.com/" credential = AzureKeyCredential("") # This client will log detailed information about its HTTP sessions, at DEBUG level form_recognizer_client = FormRecognizerClient(endpoint, credential, logging_enable=True) Similarly, ``logging_enable`` can enable detailed logging for a single operation, even when it isn't enabled for the client: .. code-block:: python poller = form_recognizer_client.begin_recognize_receipts(receipt, logging_enable=True) Next steps ---------- The following section provides several code snippets illustrating common patterns used in the Form Recognizer Python API. More sample code ^^^^^^^^^^^^^^^^ These code samples show common scenario operations with the Azure Form Recognizer client library. The async versions of the samples (the python sample files appended with ``_async``\ ) show asynchronous operations with Form Recognizer and require Python 3.5 or later. * Client authentication: `sample_authentication.py `_ (\ `async_version `_\ ) * Recognize receipts: `sample_recognize_receipts.py `_ (\ `async version `_\ ) * Recognize receipts from a URL: `sample_recognize_receipts_from_url.py `_ (\ `async version `_\ ) * Recognize content: `sample_recognize_content.py `_ (\ `async version `_\ ) * Recognize custom forms: `sample_recognize_custom_forms.py `_ (\ `async version `_\ ) * Train a model without labels: `sample_train_model_without_labels.py `_ (\ `async version `_\ ) * Train a model with labels: `sample_train_model_with_labels.py `_ (\ `async version `_\ ) * Manage custom models: `sample_manage_custom_models.py `_ (\ `async_version `_\ ) * Copy a model between Form Recognizer resources: `sample_copy_model.py `_ (\ `async_version `_\ ) Additional documentation ^^^^^^^^^^^^^^^^^^^^^^^^ For more extensive documentation on Azure Cognitive Services Form Recognizer, see the `Form Recognizer documentation `_ on docs.microsoft.com. Contributing ------------ This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit `cla.microsoft.com `_. When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. This project has adopted the `Microsoft Open Source Code of Conduct `_. For more information see the `Code of Conduct FAQ `_ or contact `opencode@microsoft.com `_ with any additional questions or comments. .. raw:: html Indices and tables ------------------ * :ref:`genindex` * :ref:`modindex` * :ref:`search` .. toctree:: :maxdepth: 5 :glob: :caption: Developer Documentation azure.ai.formrecognizer.rst