.. role:: raw-html-m2r(raw) :format: html Azure Form Recognizer client library for Python =============================================== Azure Cognitive Services Form Recognizer is a cloud service that uses machine learning to recognize text and table data from form documents. It includes the following main functionalities: * Custom models - Recognize field values and table data from forms. These models are trained with your own data, so they're tailored to your forms. * Content API - Recognize text, table structures, and selection marks, along with their bounding box coordinates, from documents. Corresponds to the REST service's Layout API. * Prebuilt models - Recognize data using the following prebuilt models * Receipt model - Recognize data from sales receipts using a prebuilt model. * Business card model - Recognize data from business cards using a prebuilt model. * Invoice model - Recognize data from invoices using a prebuilt model. `Source code `_ | `Package (PyPI) `_ | `API reference documentation `_\ | `Product documentation `_ | `Samples `_ Getting started --------------- Prerequisites ^^^^^^^^^^^^^ * Python 2.7, or 3.5 or later is required to use this package. * You must have an `Azure subscription `_ and a `Cognitive Services or Form Recognizer resource `_ to use this package. Install the package ^^^^^^^^^^^^^^^^^^^ Install the Azure Form Recognizer client library for Python with `pip `_\ : .. code-block:: bash pip install azure-ai-formrecognizer --pre .. Note: This version of the client library defaults to the v2.1-preview version of the service This table shows the relationship between SDK versions and supported API versions of the service .. list-table:: :header-rows: 1 * - SDK version - Supported API version of service * - 3.0.0 - Latest GA release (can be installed by removing the ``--pre`` flag) - 2.0 * - 3.1.0b2 - Latest release (beta) - 2.0, 2.1-preview Create a Form Recognizer resource ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Form Recognizer supports both `multi-service and single-service access `_. Create a Cognitive Services resource if you plan to access multiple cognitive services under a single endpoint/key. For Form Recognizer access only, create a Form Recognizer resource. You can create the resource using **Option 1:** `Azure Portal `_ **Option 2:** `Azure CLI `_. Below is an example of how you can create a Form Recognizer resource using the CLI: .. code-block:: bash # Create a new resource group to hold the form recognizer resource - # if using an existing resource group, skip this step az group create --name my-resource-group --location westus2 .. code-block:: bash # Create form recognizer az cognitiveservices account create \ --name form-recognizer-resource \ --resource-group my-resource-group \ --kind FormRecognizer \ --sku F0 \ --location westus2 \ --yes Authenticate the client ^^^^^^^^^^^^^^^^^^^^^^^ In order to interact with the Form Recognizer service, you will need to create an instance of a client. An **endpoint** and **credential** are necessary to instantiate the client object. Looking up the endpoint ~~~~~~~~~~~~~~~~~~~~~~~ You can find the endpoint for your Form Recognizer resource using the `Azure Portal `_ or `Azure CLI `_\ : .. code-block:: bash # Get the endpoint for the form recognizer resource az cognitiveservices account show --name "resource-name" --resource-group "resource-group-name" --query "properties.endpoint" Get the API key ~~~~~~~~~~~~~~~ The API key can be found in the Azure Portal or by running the following Azure CLI command: ``az cognitiveservices account keys list --name "resource-name" --resource-group "resource-group-name"`` Create the client with AzureKeyCredential ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To use an `API key `_ as the ``credential`` parameter, pass the key as a string into an instance of `AzureKeyCredential `_. .. code-block:: python from azure.core.credentials import AzureKeyCredential from azure.ai.formrecognizer import FormRecognizerClient endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) Create the client with an Azure Active Directory credential ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``AzureKeyCredential`` authentication is used in the examples in this getting started guide, but you can also authenticate with Azure Active Directory using the `azure-identity `_ library. Note that regional endpoints do not support AAD authentication. Create a `custom subdomain `_ name for your resource in order to use this type of authentication. To use the `DefaultAzureCredential `_ type shown below, or other credential types provided with the Azure SDK, please install the ``azure-identity`` package: ``pip install azure-identity`` You will also need to `register a new AAD application and grant access `_ to Form Recognizer by assigning the ``"Cognitive Services User"`` role to your service principal. Once completed, set the values of the client ID, tenant ID, and client secret of the AAD application as environment variables: ``AZURE_CLIENT_ID``\ , ``AZURE_TENANT_ID``\ , ``AZURE_CLIENT_SECRET``. .. code-block:: python from azure.identity import DefaultAzureCredential from azure.ai.formrecognizer import FormRecognizerClient credential = DefaultAzureCredential() form_recognizer_client = FormRecognizerClient( endpoint="https://.cognitiveservices.azure.com/", credential=credential ) Key concepts ------------ FormRecognizerClient ^^^^^^^^^^^^^^^^^^^^ ``FormRecognizerClient`` provides operations for: * Recognizing form fields and content using custom models trained to recognize your custom forms. These values are returned in a collection of ``RecognizedForm`` objects. * Recognizing common fields from the following form types using prebuilt models. These fields and metadata are returned in a collection of ``RecognizedForm`` objects. * Sales receipts. See fields found on a receipt `here `_. * Business cards. See fields found on a business card `here `_. * Invoices. See fields found on an invoice `here `_. * Recognizing form content, including tables, lines, words, and selection marks, without the need to train a model. Form content is returned in a collection of ``FormPage`` objects. Sample code snippets are provided to illustrate using a FormRecognizerClient :raw-html-m2r:`here`. FormTrainingClient ^^^^^^^^^^^^^^^^^^ ``FormTrainingClient`` provides operations for: * Training custom models without labels to recognize all fields and values found in your custom forms. A ``CustomFormModel`` is returned indicating the form types the model will recognize, and the fields it will extract for each form type. See the `service documentation `_ for a more detailed explanation. * Training custom models with labels to recognize specific fields, selection marks, and values you specify by labeling your custom forms. A ``CustomFormModel`` is returned indicating the fields the model will extract, as well as the estimated accuracy for each field. See the `service documentation `_ for a more detailed explanation. * Managing models created in your account. * Copying a custom model from one Form Recognizer resource to another. * Creating a composed model from a collection of existing trained models with labels. Please note that models can also be trained using a graphical user interface such as the `Form Recognizer Labeling Tool `_. Sample code snippets are provided to illustrate using a FormTrainingClient :raw-html-m2r:`here`. Long-Running Operations ^^^^^^^^^^^^^^^^^^^^^^^ Long-running operations are operations which consist of an initial request sent to the service to start an operation, followed by polling the service at intervals to determine whether the operation has completed or failed, and if it has succeeded, to get the result. Methods that train models, recognize values from forms, or copy/compose models are modeled as long-running operations. The client exposes a ``begin_`` method that returns an ``LROPoller`` or ``AsyncLROPoller``. Callers should wait for the operation to complete by calling ``result()`` on the poller object returned from the ``begin_`` method. Sample code snippets are provided to illustrate using long-running operations :raw-html-m2r:`below`. Examples -------- The following section provides several code snippets covering some of the most common Form Recognizer tasks, including: * :raw-html-m2r:`Recognize Forms Using a Custom Model` * :raw-html-m2r:`Recognize Content` * :raw-html-m2r:`Recognize Receipts` * :raw-html-m2r:`Recognize Business Cards` * :raw-html-m2r:`Recognize Invoices` * :raw-html-m2r:`Train a Model` * :raw-html-m2r:`Manage Your Models` Recognize Forms Using a Custom Model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Recognize name/value pairs and table data from forms. These models are trained with your own data, so they're tailored to your forms. For best results, you should only recognize forms of the same form type that the custom model was trained on. .. code-block:: python from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) model_id = "" with open("", "rb") as fd: form = fd.read() poller = form_recognizer_client.begin_recognize_custom_forms(model_id=model_id, form=form) result = poller.result() for recognized_form in result: print("Form type: {}".format(recognized_form.form_type)) print("Form type confidence: {}".format(recognized_form.form_type_confidence)) print("Form was analyzed using model with ID: {}".format(recognized_form.model_id)) for name, field in recognized_form.fields.items(): print("Field '{}' has label '{}' with value '{}' and a confidence score of {}".format( name, field.label_data.text if field.label_data else name, field.value, field.confidence )) Alternatively, a form URL can also be used to recognize custom forms using the ``begin_recognize_custom_forms_from_url`` method. The ``_from_url`` methods exist for all the recognize methods. .. code-block:: form_url = "" poller = form_recognizer_client.begin_recognize_custom_forms_from_url(model_id=model_id, form_url=form_url) result = poller.result() Recognize Content ^^^^^^^^^^^^^^^^^ Recognize text, selection marks, and table structures, along with their bounding box coordinates, from documents. .. code-block:: python from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) with open("", "rb") as fd: form = fd.read() poller = form_recognizer_client.begin_recognize_content(form) form_pages = poller.result() for content in form_pages: for table in content.tables: print("Table found on page {}:".format(table.page_number)) print("Table location {}:".format(table.bounding_box)) for cell in table.cells: print("Cell text: {}".format(cell.text)) print("Location: {}".format(cell.bounding_box)) print("Confidence score: {}\n".format(cell.confidence)) if content.selection_marks: print("Selection marks found on page {}:".format(content.page_number)) for selection_mark in content.selection_marks: print("Selection mark is '{}' within bounding box '{}' and has a confidence of {}".format( selection_mark.state, selection_mark.bounding_box, selection_mark.confidence )) Recognize Receipts ^^^^^^^^^^^^^^^^^^ Recognize data from sales receipts using a prebuilt model. Receipt fields recognized by the service can be found `here `_. .. code-block:: python from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) with open("", "rb") as fd: receipt = fd.read() poller = form_recognizer_client.begin_recognize_receipts(receipt) result = poller.result() for receipt in result: for name, field in receipt.fields.items(): if name == "Items": print("Receipt Items:") for idx, items in enumerate(field.value): print("...Item #{}".format(idx+1)) for item_name, item in items.value.items(): print("......{}: {} has confidence {}".format(item_name, item.value, item.confidence)) else: print("{}: {} has confidence {}".format(name, field.value, field.confidence)) Recognize Business Cards ^^^^^^^^^^^^^^^^^^^^^^^^ Recognize data from business cards using a prebuilt model. Business card fields recognized by the service can be found `here `_. .. code-block:: python from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) with open("", "rb") as fd: business_card = fd.read() poller = form_recognizer_client.begin_recognize_business_cards(business_card) result = poller.result() for business_card in result: for name, field in business_card.fields.items(): if name == "ContactNames": print("ContactNames:") for items in field.value: for item_name, item in items.value.items(): print("...{}: {} has confidence {}".format(item_name, item.value, item.confidence)) else: for item in field.value: print("{}: {} has confidence {}".format(item.name, item.value, item.confidence)) Recognize Invoices ^^^^^^^^^^^^^^^^^^ Recognize data from invoices using a prebuilt model. Invoice fields recognized by the service can be found `here `_. .. code-block:: python from azure.ai.formrecognizer import FormRecognizerClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_recognizer_client = FormRecognizerClient(endpoint, credential) with open("", "rb") as fd: invoice = fd.read() poller = form_recognizer_client.begin_recognize_invoices(invoice) result = poller.result() for invoice in result: for name, field in invoice.fields.items(): print("{}: {} has confidence {}".format(name, field.value, field.confidence)) Train a model ^^^^^^^^^^^^^ Train a custom model on your own form type. The resulting model can be used to recognize values from the types of forms it was trained on. Provide a container SAS URL to your Azure Storage Blob container where you're storing the training documents. If training files are within a subfolder in the container, use the `prefix `_ keyword argument to specify under which folder to train. More details on setting up a container and required file structure can be found in the `service documentation `_. .. code-block:: python from azure.ai.formrecognizer import FormTrainingClient from azure.core.credentials import AzureKeyCredential endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_training_client = FormTrainingClient(endpoint, credential) container_sas_url = "" # training documents uploaded to blob storage poller = form_training_client.begin_training( container_sas_url, use_training_labels=False, model_name="my first model" ) model = poller.result() # Custom model information print("Model ID: {}".format(model.model_id)) print("Model name: {}".format(model.model_name)) print("Is composed model?: {}".format(model.properties.is_composed_model)) print("Status: {}".format(model.status)) print("Training started on: {}".format(model.training_started_on)) print("Training completed on: {}".format(model.training_completed_on)) print("\nRecognized fields:") for submodel in model.submodels: print( "The submodel with form type '{}' and model ID '{}' has recognized the following fields: {}".format( submodel.form_type, submodel.model_id, ", ".join( [ field.label if field.label else name for name, field in submodel.fields.items() ] ), ) ) # Training result information for doc in model.training_documents: print("Document name: {}".format(doc.name)) print("Document status: {}".format(doc.status)) print("Document page count: {}".format(doc.page_count)) print("Document errors: {}".format(doc.errors)) Manage Your Models ^^^^^^^^^^^^^^^^^^ Manage the custom models attached to your account. .. code-block:: python from azure.ai.formrecognizer import FormTrainingClient from azure.core.credentials import AzureKeyCredential from azure.core.exceptions import ResourceNotFoundError endpoint = "https://.api.cognitive.microsoft.com/" credential = AzureKeyCredential("") form_training_client = FormTrainingClient(endpoint, credential) account_properties = form_training_client.get_account_properties() print("Our account has {} custom models, and we can have at most {} custom models".format( account_properties.custom_model_count, account_properties.custom_model_limit )) # Here we get a paged list of all of our custom models custom_models = form_training_client.list_custom_models() print("We have models with the following ids: {}".format( ", ".join([m.model_id for m in custom_models]) )) # Replace with the custom model ID from the "Train a model" sample model_id = "" custom_model = form_training_client.get_custom_model(model_id=model_id) print("Model ID: {}".format(custom_model.model_id)) print("Model name: {}".format(custom_model.model_name)) print("Is composed model?: {}".format(custom_model.properties.is_composed_model)) print("Status: {}".format(custom_model.status)) print("Training started on: {}".format(custom_model.training_started_on)) print("Training completed on: {}".format(custom_model.training_completed_on)) # Finally, we will delete this model by ID form_training_client.delete_model(model_id=custom_model.model_id) try: form_training_client.get_custom_model(model_id=custom_model.model_id) except ResourceNotFoundError: print("Successfully deleted model with id {}".format(custom_model.model_id)) Troubleshooting --------------- General ^^^^^^^ Form Recognizer client library will raise exceptions defined in `Azure Core `_. Logging ^^^^^^^ This library uses the standard `logging `_ library for logging. Basic information about HTTP sessions (URLs, headers, etc.) is logged at ``INFO`` level. Detailed ``DEBUG`` level logging, including request/response bodies and **unredacted** headers, can be enabled on the client or per-operation with the ``logging_enable`` keyword argument. See full SDK logging documentation with examples `here `_. Optional Configuration ^^^^^^^^^^^^^^^^^^^^^^ Optional keyword arguments can be passed in at the client and per-operation level. The azure-core `reference documentation `_ describes available configurations for retries, logging, transport protocols, and more. Next steps ---------- The following section provides several code snippets illustrating common patterns used in the Form Recognizer Python API. More sample code ^^^^^^^^^^^^^^^^ These code samples show common scenario operations with the Azure Form Recognizer client library. * Client authentication: `sample_authentication.py `_ * Recognize receipts: `sample_recognize_receipts.py `_ * Recognize receipts from a URL: `sample_recognize_receipts_from_url.py `_ * Recognize business cards: `sample_recognize_business_cards.py `_ * Recognize invoices: `sample_recognize_invoices.py `_ * Recognize content: `sample_recognize_content.py `_ * Recognize custom forms: `sample_recognize_custom_forms.py `_ * Train a model without labels: `sample_train_model_without_labels.py `_ * Train a model with labels: `sample_train_model_with_labels.py `_ * Manage custom models: `sample_manage_custom_models.py `_ * Copy a model between Form Recognizer resources: `sample_copy_model.py `_ * Create a composed model from a collection of models trained with labels: `sample_create_composed_model.py `_ Async APIs ^^^^^^^^^^ This library also includes a complete async API supported on Python 3.5+. To use it, you must first install an async transport, such as `aiohttp `_. Async clients are found under the ``azure.ai.formrecognizer.aio`` namespace. * Client authentication: `sample_authentication_async.py `_ * Recognize receipts: `sample_recognize_receipts_async.py `_ * Recognize receipts from a URL: `sample_recognize_receipts_from_url_async.py `_ * Recognize business cards: `sample_recognize_business_cards_async.py `_ * Recognize invoices: `sample_recognize_invoices_async.py `_ * Recognize content: `sample_recognize_content_async.py `_ * Recognize custom forms: `sample_recognize_custom_forms_async.py `_ * Train a model without labels: `sample_train_model_without_labels_async.py `_ * Train a model with labels: `sample_train_model_with_labels_async.py `_ * Manage custom models: `sample_manage_custom_models_async.py `_ * Copy a model between Form Recognizer resources: `sample_copy_model_async.py `_ * Create a composed model from a collection of models trained with labels: `sample_create_composed_model_async.py `_ Additional documentation ^^^^^^^^^^^^^^^^^^^^^^^^ For more extensive documentation on Azure Cognitive Services Form Recognizer, see the `Form Recognizer documentation `_ on docs.microsoft.com. Contributing ------------ This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit `cla.microsoft.com `_. When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. This project has adopted the `Microsoft Open Source Code of Conduct `_. For more information see the `Code of Conduct FAQ `_ or contact `opencode@microsoft.com `_ with any additional questions or comments. .. raw:: html Indices and tables ------------------ * :ref:`genindex` * :ref:`modindex` * :ref:`search` .. toctree:: :maxdepth: 5 :glob: :caption: Developer Documentation azure.ai.formrecognizer.rst