Searching document text at scale using Azure Cognitive Search

This blog post is accompanied by another post entitled Creating custom skills in Azure Cognitive Search using Azure ML.

This is the second of the two posts. In the first post we explore the creation of a custom skill for use in enriching the document index we’ll be creating here.

Azure cognitive search is a Lucene-based search PaaS service available from Microsoft Azure.

In this post we’ll go through the process of:

• Creating an Azure Storage Account
• Uploading documents to our Azure Storage Account
• Creating an Azure Cognitive Search instance
• Connecting the Azure Cognitive Search instance to our data source
• Connecting the Azure Cognitive Search instance to Cognitive Services
• Defining a skillset including our custom skill
• Indexing our documents
• Querying Azure Search
• Formatting search highlights

The workflow for Azure Cognitive Search looks like the following diagram:

The tools we’ll be using in this post are the Azure CLI and Python to make REST requests to our Cognitive Search instance.

Quick disclaimer: At the time of writing, I am currently a Microsoft Employee

Example Use Case

The example use case to be used here is that we’ll be uploading PDF files, having Azure use the OCR service from Azure Cognitive Services to insert any non-machine readable text, and making the resulting text searchable using Azure Cognitive Search.

The PDF files to be used in this case are a set of 10 PDF files from the year 1980 from the Open Access Journal Nucleic Acids Research (NAR). They can be found here.

Although only 10 PDF files are used here, this can be done at a much larger scale and Azure Cognitive Search supports a range of other file formats including: Microsoft Office (DOCX/DOC, XSLX/XLS, PPTX/PPT, MSG), HTML, XML, ZIP, and plain text files (including JSON).

For reference, the author was previously a Biochemist, with 3 papers published in NAR, though not quite in 1980 – Nucleic Acids Research tends to focus on the field of molecular biology, particularly in relation to DNA and RNA, and proteins and other molecules that interact with these nucleic acids.

An example of what one of our files looks like can be seen here:

Creating an Azure Storage Account

We’ll start by creating an Azure Storage Account in which to store our documents.

Using the Azure CLI, create a resource group (if not already created), I’ve named mine azure-search-nar-demo:

az group create --name azure-search-nar-demo --location westeurope

Then, again using the Azure CLI, create a storage account, this one is named narsearchdemostorage:

az storage account create --name narsearchdemostorage --resource-group azure-search-nar-demo --location westeurope

We’ll need the connection string for this account, to retrieve it for this account run the following:

az storage account show-connection-string --name narsearchdemostorage

Uploading PDF files to Azure Storage Account

We’ll be using the following two external python packages in this post, so I’d recommend pip installing them into a python virtual environment:

pip install azure-storage-blob
pip install requests

Let’s go ahead and import the python modules we’ll need for this section:

Now we’ll instantiate the blob service client using the connection string we got from the Azure CLI in the previous section:

Within this storage account, we’ll need to create a BLOB container, this is done as follows:

Before we upload our files, let’s list the directory and take a look at what files we’ll be uploading:

We can now iterate through these files and upload each of them to our storage account (it may be easier for larger numbers of files to use the AzCopy tool or Azure Storage Explorer).

To check our files have uploaded, we can list the files in the container:

Create Azure Cognitive Search Service

We’ll now create our Azure Cognitive Search service, I’ve called mine nar-demo-search – Using Azure CLI run:

az search service create --name nar-demo-search --resource-group azure-search-nar-demo --location westeurope --sku standard

SKU options can be found here.

This can take a while so go and grab a coffee while this deploys.

Once deployed, we’ll need the admin key for this Azure Cognitive Search service instance, to access this you can run using Azure CLI:

az search admin-key show --service-name nar-demo-search --resource-group azure-search-nar-demo

Connect Azure Cognitive Search Service to BLOB storage

In the cell below, we’ll import modules and define variables that we’ll need for each of the Azure Cognitive Search Service REST requests for the rest of this blog post.

For each request to Azure Cognitive Search, you must provide an API Version, in this post, we’re using the latest stable version, which is "2019-05-06".

Now we can connect to our data source, we’ll need to provide the Azure Storage Account connection string we previously used to connect to our Storage account and we’ll need to give this data source a name.

We also provide the container in which our data is stored.

Connect Azure Cognitive Search Service to Cognitive Services

Next we’re going to define the AI skillset which will be used to enrich our search index.

But first, in order to do this, it’s advisable to create an Azure Cognitive Services instance, otherwise your AI enrichment capabilities will be severely limited in scope. This Azure Cognitive Services instance must be in the same region as your Azure Cognitive Search instance, and can be created from the CLI, I’ve named mine nar-demo-cognitive-services:

az cognitiveservices account create --name nar-demo-cognitive-services --kind CognitiveServices --sku S0 --resource-group azure-search-nar-demo --location westeurope

We’ll need the account key for this Azure Cognitive Services instance in order to connect our Azure Cognitive Search instance to it, we can retrieve the key using the Azure CLI:

az cognitiveservices account keys list --name nar-demo-cognitive-services --resource-group azure-search-nar-demo

Create Azure Cognitive Search Skillset

To enrich our Azure Cognitive Search Index with an AI skillset, we’ll need to define a skillset.

We’ll be using a combination of skills that utilise Azure Cognitive Services and our own custom skill deployed in Azure Machine Learning. The skills we’ll be using are:

• OCR to extract text from image
• Merge text extracted from OCR into the correct place in documents
• Detect language
• Split text into pages if not already done so
• Key phrase extraction (has a maximum character limit so requires text to be split into pages)
• Our Custom Skill to extract genetic codes.

The skillset definition is a JSON object and each of the skills defined take one or more fields as input and provide one or more fields as output.

We’ll need to provide the Cognitive Search API with the key to our Cognitive Services account in order to use it, so make sure you put this in your skillset definition.

We’ll also need the URI and key for our Azure Cognitive Search Skill API that we previously deployed in Azure Machine Learning, so make sure to update the skillset definition below too.

The skillset is created through a PUT request to our Azure Cognitive Search skillsets REST endpoint.

The data above is serialised to a JSON string before being provided as the body of the request:

Create Azure Cognitive Search Index

The index is the definition of fields that will be returned, as well as metadata such as the data type of this field, whether it is a key or not, and whether it is searchable, filterable, facetable or sortable.

We’ll return the id, metadata_storage_name, content, languageCode, keyPhrases, genetic_codes, and merged_text field and make all of them searchable.

Just as with our skillset, the index is created through a PUT request to our Azure Cognitive Search indexes REST endpoint.

The data above is serialised to a JSON string before being provided as the body of the request:

Create Azure Cognitive Search Indexer

We’ll now create our indexer, which runs through our data and indexes it as we’ve defined above using the data source, index and skillset given.

We can set maxFailedItems to maxFailedItemsPerBatch to -1 if we want the indexer to continue through until reaching the end of all the documents regardless of how many failures it encounters.

The storage path of the file within the storage container will be base 64 encoded and act as a key as it will be unique for each document, regardless of whether two files have the same filename.

You may be sensing a theme now to creating associated resources for our Azure Cognitive Search instance. Just as with our skillset and index, the indexer is created through a PUT request to our Azure Cognitive Search indexers REST endpoint.

The data above is serialised to a JSON string before being provided as the body of the request. This will kick off the indexing on our PDF files.

We have not defined a schedule in our indexer so it will just run once but if you wish to provide a schedule that will check for updates to the index you can, for example, every 2 hours would be:
"schedule" : { "interval" : "PT2H" }
And 5 minutes, the shortest interval, would be:
"schedule" : { "interval" : "PT5M" }

To check on the status of our indexer, you can run the following cell.

The JSON response has more information for if you want to know how many files the indexer has completed and how many is left to go.

Querying the Azure Cognitive Search Index

Now that our documents have been indexed, we can start to query our Azure Cognitive Search index.

I mentioned that the journal NAR is focused on the research of nucleic acids DNA and RNA, so we should expect that if we were to search for, say, the term “RNA” we should get back most, if not all, of our documents.

The 200 status code is a good start. To get the search results from our response, we must call the json() method of our response object to de-serialise the JSON string object that is returned in the body of our response.

This object is a dict with 3 keys:

• @odata.context – The index and documents that were searched
• @odata.count – The count of documents returned from our index search query
• value – The output fields of the index search query

We can see in the cells above and below that we are, indeed, returned all 10 documents by querying our index with the word “RNA”.

For each item in value, we are returned the fields that we defined in our index, as well as a search score:

We can take a look at the results of our search by looking at the PDF name and search score.

By default, the output of the query is ordered by search score and we have not defined any weightings to fields but this is something that can also be defined:

In the cell below, we take a look at the beginning of the merged text of the highest scoring search result. Given that the name of this article is “Evidence for the role of double-helical structures in the maturation of Sinian Virus-40 messenger RNA”, it’s unsurprising that this is the highest scoring search result when searching for “RNA”.

Let’s now take a look at a query that won’t just return every document that we uploaded.

Transcription is the mechanism by which DNA is encoded into RNA as the initial step in gene expression, ready to be translated into proteins.

Let’s do a search for “transcription” and see how many results our query returns.

In the following cell, we find that the term transcription is found in 4 out of our 10 documents.

When looking at our highest result, we can be buoyed once more by the fact that the highest scoring search result is likely to be highly related to transcription – this journal article is about the regions of the ovalbumin gene that are responsible for controlling gene expression and is led by noted Molecular Geneticist Pierre Chambon, whose work has focused on gene transcription.

Custom Skill Output

In the other blog post Creating custom skills in Azure Cognitive Search using Azure ML Service we defined a custom skill to extract genetic codes from journal articles when the index is created.

The genetic codes are returned in our search results.

We can provide a search query to search within collections using e.g. search=collection_name/any(x: x eq 'foo') to search for foo within the collection collection_name.

So if we search for one of the genetic codes above, this journal article will be top of our search results.

Pagination of Search Results

As we might not want all results all of the time from our search, and indeed by default only the first 50 results will be returned, we might want to paginate our results, to allow users to explore results in batches.

Let’s say we want to search for RNA, which we already know will find 10 results, but we only want to display 5 results on a page. We can do this by supplying to our URL a $top parameter.

That’s perfect for our first page, but what about the second page? We need a way of skipping the first 5 results and showing the next 5 highest scoring results.

We can do this by supplying both $top and $skip parameters to our URL.

Article Highlights from Search Results

One of my favourite features of Azure Cognitive Search is the ability to highlight parts of the document that are relevant to our search results. Combined with the above, this really helps turn our API into a proper search engine.

We can request highlights on a particular field by adding the highlight parameter to our URL as below. Let’s highlight the results we had from our search for “transcription”

This will extract relevant parts of the journal article regarding “transcription”.

We can see in the cell below, which shows the highlights from the search result, that this will return a list of highlights from the article and even provides HTML em tags for the word transcription.

If we’re using a jupyter notebook, we can display these highlights with the word transcription emphasised through italicisation.

But that’s not all, if we wanted to supply our own tags to, for example, highlight instead of italicise these highlights we can supply our own HTML tags to surround the query phrase.

In this case we’re providing span tags with some inline CSS.

Now when we display our search results, the output looks a little bit more like that of a regular search engine.

Feel free to explore your Azure Cognitive Search endpoints and don’t forget to delete your resource group when you’re finished exploring.