Introduction to QnA Maker

Suppose you have a FAQ page that has a lot of data and want to use that as a first line of customer service support for a chat bot on your main page. How can you integrate this with minimal effort? Enter Microsoft Q&A Maker.

For the video verison of this post, check below.

What is Microsoft QnA Maker

From the official docs it states that...

QnA Maker is a cloud-based Natural Language Processing (NLP) service that allows you to create a natural conversational layer over your data.

So, basically, this service allows you to creates question and answering based on the data that you have.

Creating the Azure Resource

First thing, like for all Cognitive Services items, we need to create the Azure resource. When creating a new resource, you can search for "QnA" and select "QnA Maker".

After clicking "Create" on that, we're on a screen where we have to enter a few things. First, we will supply the subscription, resource group, name, and pricing tier. Note that this does have a free tier so this can be used for proof of concepts or to simply try out the service to see if it meets your needs.

Next, it will ask for details for Azure Search. This is used to host the data that you give it for the QnA maker and Azure Search is used for that. Only the location and pricing tier is needed for this. This has a free tier, as well.

Last, it will ask details for an App Service. This is used to host the runtime for QnA maker which will be used to make queries against your data. The app name and location is required for this.

You can optionally enable app insights for this service as well, but feel free to disable that since it's not really needed unless you are creating this for actual business purposes and want to see if anything goes wrong.

With all that, we can click the "Review and Create" button to create and deploy the resources.

Creating a QnA Knowledge Base

With the resource created we can now go to it. Per usual with the Cognitive Services items, we have a "Keys and Endpoint" item on the left where we can get the key and endpoint. This will be used later when using the API.

But first, we need to create our QnA Knowledge Base and to do that we need to go to the "Overview" tab on the left navigation. A bit down it has a link where we can go to the QnA portal to create a new knowledge base.

We can skip step one since we already created the Azure resource. In step two, we will connect the QnA portal to our Azure resource. Simply give it the subscription and the Azure service name. Luckily, all of this pre-populates so they are all dropdowns.

In step three, we will give our knowledge base a name. We'll name it "ikea" since we will use the Ikea FAQ to populate the knowledge base.

Step four is where we'll populate the knowledge base. If you already have a FAQ on your website you can put the URL in. Since I'm using the Ikea FAQ we can do that. You can also add a file for this. If you have neither, you can leave this blank and fill out the questions and answers manually on the next page.

Below this you can customize your bot with a chit-chat. This just helps give your QnA bot a bit more personality based on what you select. Here's a screenshot of an example from the docs of each of the chit-chat items that you can choose from.

For step five, we can create our knowledge base by clicking on the button.

Training and Testing

Once we have our knowledge base created we can look see that it read in the Ikea FAQ quite well. To see just how well QnA Maker is, we can instantly click on the "Save and train" button to train a model on our data.

Once that finishes we can click on the "Test" button to give the model a test. This is an integrated chat bot where we can ask it questions and will receive answers based on the model.

So we can ask "What is the return policy?" and QnA Maker will give us the best answer based on our data.

Early on we get some good results from QnA Maker. But what if we want to add a new pair?

Adding a New QnA Pair

If we want to add new QnA pairs to our existing knowledge base, just click on the Add QnA Pair button.

We can add an alternate phrasing such as "This is a question". A phrasing is mostly a question that a user would input into the system that will get sent to QnA Maker. We can input an answer as well, such as "This is an answer". Notice that we have a rich text editor which can be toggled in the upper left. With this, we can add graphics, links, and emojis. Let's add a smile emoji to our answer.

Now we can click the "Save and train" button to train a new model on what we just added. We can then do another test and in put "This is a question" and we should get the answer that we put as the output.

Using the API

Before we can actually use the API we need to publish our knowledge base. Simply click the "Publish" tab near the top and then the "Publish" button.

Once that completes you can either create a bot that will use this knoweldge base, or you can use the API directly. We'll use the API and the publish page shows how you can call the API using Postman or through curl. We'll use Postman here so we can easily test the API out.

To build the URL for the API, use the "Host" item in the deployment details and then append the first item where it has "POST" after that.

And since it does say "POST" we will make this as a POST call.

Next, we need to set the authorization header. In the Authorization tab in Postman, set the type to "API Key". The "Key" item will be "Authorization" and the "Value" will be the API key which is the third part of the deployment details.

Now, we can add in the JSON body and the simplest JSON that we can add has only one item, a "question" item. And this is the prompt that a user would send to QnA Maker. Let's add the question that we added earlier.

{
 "question":"This is a question"
}

Once we hit "Send" in Postman, we will get the below response.

{
     "answers": [
     {
         "questions": [
             "This is a question"
         ],
         "answer": "This is an answer😀",
         "score": 100.0,
         "id": 176,
         "source": "Editorial",
         "isDocumentText": false,
         "metadata": [],
         "context": {
             "isContextOnly": false,
             "prompts": []
         }

    }
 ],
 "activeLearningEnabled": false
}

The main part to notice here is the "answer", which is what we expect to get back.


Hopefully, this showed how useful the QnA Maker can be, especially if you already have a FAQ page with questions and answers. With QnA Maker it can be turned into a chat bot or any other automation tool where users or customers may need to ask questions.

Use Bing Image Search to Get Training Image Data

When going through the FastAI book, Deep Learning for Coders, I noticed that in one of the early chapters they mention using the Bing Image Search API to retrieve images for training data. While they have a nice wrapper for the API, I thought I'd dive into the API as well and use it to build my own way to download training image data.

Let's suppose we need to make an image classification model to determine what kind of car is in an image. We'd need quite a bit of different images for this model, so let's use the Bing Image Search to gather images of the Aston-Martin car so we can start getting our data.

Check out the below for a video version of this post.

Before going into the technical side of Bing Image Search, let's go over why use this in the first place. Why not just download the images ourselves?

Bing Images Search has a few features in it that we can utilize in our code when getting our images. Some of these features are important to take into account.

Automation

I'll be honest, I'm lazy and if I can script something to do a task for me then I'll definitely spend the time to build the script rather than do the task manually. Rather than manually finding images and downloading them, we can use the Bing Image Search API to do this for us.

Image License

We can't always just take an image from a website and use it however we want. A lot of images that are online are copyrighted and if we don't have a license to use the copyright we are actually in violation of the creator's copyright. If they find out we use their image without a license or permission then they can, more than likely, take legal action against us.

However, with Bing Image Search, we have an option to specify what license the images has that get returned to us. We can do this with the licenseType query parameter in our API call. This utilizes Creative Commons licenses. We can specify exactly what type of license our images has. We can specify that want images that are public where the copyright is fully waived, which is what we will do. There are many Creative Commons license types that the Bing Image Search supports and there's a full list here.

Image Type

There are quite a few images types that we could download from Bing Image Search. For our purposes, though, we only want photos of Aston Martin cars. Due to that, we can specify the image type in our API calls to just photo. If we don't specify this we could get back animated GIFs, clip art, or drawings of Aston Martin cars.

Safe Content

When downloading images from the internet you never really know what you're going to get. Bing Image Search can help ease that worry by specifying that you want only safe content to be returned.

Bing can do this filtering for us so we don't have to worry about it when we do our API call. This is one less thing we have to worry about and, because it's the internet, it's definitely something to worry about when download images.

Create Azure Resource

Before we can use the Bing Image Search API we need to create the resource for it. In the Azure Portal create a new resource and search for "Bing Search". Then, click on the "Bing Search v7" resource to create it.

When creating the resource give it a name, what resource group it will be under, and what subscription it will be under. For the pricing tier, it does have a free tier to allow you to give the service a try for evaluation or for a proof of concept. Once that is complete, click "Create".

When that completes deployment, we can explore a bit on the resource page. One thing to note is that there are a few things we can look at here. There's a "Try me" tab where we can try the Bing Search API and see what results we get. There is some sample code to see real quick how to use the Bing Search API. And there are a lot of tutorials that we can look at if we want to look at something more specific, such as the image or video search APIs.

Retrieve Key and Endpoint

To use the API in our code we will need the API key and the endpoint to call. There are a couple of ways we can get to it. First, on the "Overview" page of the resource there's a link that says to "click here to manage keys". Clicking that will take you to another page where you can get the API keys and the endpoint URL.

You can also click on the "Keys and Endpoint" section on the left navigation.

Now save the API key and the endpoint since we'll need those to access the API in the code.

Using the API

Now we get to the fun stuff where we can get into some code. I'll be using Python, but you're very welcome to use the language of your choice since this is a simple API call. I'm also using Azure ML since it's very easy to get a Jupyter Lab instance running plus most machine learning and data science packages already installed.

Imports

First, we need to import some modules. We have four that we will need to import.

  • JSON: This will be used to read in a config file for the API key and endpoint
  • Requests: Will be used to make the API calls. This is pre-installed in an Azure ML Jupyter instance, so you may need to run pip install requests if you are using another envrionment.
  • Time: Used to delay API calls so the server doesn't get hit too much by requests.
  • OS: Used to saved and help clean image data on the local machine.
  • PPrint: Used to format JSON when printing.
import json
import requests
import time
import os
import pprint

The API Call

Now, we can start building and making the API call to get the image data.

Building the Endpoint

To start building the call, we need to get the API key which is kept in a JSON file for security reasons. We'll use the open method to open the file to be able to read it and use the json module to load the JSON file. This creates a dictionary where the JSON keys are the key names of the dictionary where you can get the values.

config = json.load(open("config.json"))

api_key = config["apiKey"]

Now that we have the API key we can build up the URL to make the API call. We can use the endpoint that we got from the Azure Portal and help build up the URL.

endpoint = "https://api.bing.microsoft.com/"

With the endpoint, we have to add some to it to tell it that we want the Image Search API. To learn more about the exact endpoints we're using here, this doc has a lot of good information.

url = f"{endpoint}v7.0/images/search"

Building the Headers and Query Parameters

Some more information we need to add to our call are the headers and the query parameters. The headers is where we supply the API key and the query parameters detail what images we want to return.

Requests makes it easy to specify the headers, which is done as a dictionary. We need to supply the Ocp-Apim-Subscription-Key header for the API key.

headers = { "Ocp-Apim-Subscription-Key": api_key }

The query parameters are also done as a dictionary. We'll supply the license, image type, and safe search parameters here. Those are optional parameters, but the q parameter is required which is what query we want to use to search for images. For our query here, we'll search for aston martin cars.

params = {
    "q": "aston martin", 
    "license": "public", 
    "imageType": "photo",
    "safeSearch": "Strict",
}

Making the API Call

With everything ready, we can now make the API call and get the results. With requests we can just call the get method. In there we pass in the URl, the headers, and the parameters. We use the raise_for_status method to throw an exception if the status code isn't successful. Then, we get the JSON of the response and store that into a variable. Finally, we use the pretty print method to print the JSON response.

response = requests.get(url, headers=headers, params=params)
response.raise_for_status()

result = response.json()

pprint.pprint(result)

And here's a snapshot of the response. There's quite a bit here but we'll break it down some later in this post.

{'_type': 'Images',
 'currentOffset': 0,
 'instrumentation': {'_type': 'ResponseInstrumentation'},
 'nextOffset': 38,
 'totalEstimatedMatches': 475,
 'value': [{'accentColor': 'C6A105',
            'contentSize': '1204783 B',
            'contentUrl': '[https://www.publicdomainpictures.net/pictures/380000/velka/aston-martin-car-1609287727yik.jpg](https://www.publicdomainpictures.net/pictures/380000/velka/aston-martin-car-1609287727yik.jpg)',
            'creativeCommons': 'PublicNoRightsReserved',
            'datePublished': '2021-02-06T20:45:00.0000000Z',
            'encodingFormat': 'jpeg',
            'height': 1530,
            'hostPageDiscoveredDate': '2021-01-12T00:00:00.0000000Z',
            'hostPageDisplayUrl': '[https://www.publicdomainpictures.net/view-image.php?image=376994&picture=aston-martin-car](https://www.publicdomainpictures.net/view-image.php?image=376994&picture=aston-martin-car)',
            'hostPageFavIconUrl': '[https://www.bing.com/th?id=ODF.lPqrhQa5EO7xJHf8DMqrJw&pid=Api](https://www.bing.com/th?id=ODF.lPqrhQa5EO7xJHf8DMqrJw&pid=Api)',
            'hostPageUrl': '[https://www.publicdomainpictures.net/view-image.php?image=376994&picture=aston-martin-car](https://www.publicdomainpictures.net/view-image.php?image=376994&picture=aston-martin-car)',
            'imageId': '38DBFEF37523B232A6733D7D9109A21FCAB41582',
            'imageInsightsToken': 'ccid_WTqn9r3a*cp_74D633ADFCF41C86F407DFFCF0DEC38F*mid_38DBFEF37523B232A6733D7D9109A21FCAB41582*simid_608053462467504486*thid_OIP.WTqn9r3aKv5TLZxszieEuQHaF5',
            'insightsMetadata': {'availableSizesCount': 1,
                                 'pagesIncludingCount': 1},
            'isFamilyFriendly': True,
            'name': 'Aston Martin Car Free Stock Photo - Public Domain '
                    'Pictures',
            'thumbnail': {'height': 377, 'width': 474},
            'thumbnailUrl': '[https://tse2.mm.bing.net/th?id=OIP.WTqn9r3aKv5TLZxszieEuQHaF5&pid=Api](https://tse2.mm.bing.net/th?id=OIP.WTqn9r3aKv5TLZxszieEuQHaF5&pid=Api)',
            'webSearchUrl': '[https://www.bing.com/images/search?view=detailv2&FORM=OIIRPO&q=aston+martin&id=38DBFEF37523B232A6733D7D9109A21FCAB41582&simid=608053462467504486](https://www.bing.com/images/search?view=detailv2&FORM=OIIRPO&q=aston+martin&id=38DBFEF37523B232A6733D7D9109A21FCAB41582&simid=608053462467504486)',
            'width': 1920}]

A few things to note from the response:

  • nextOffset: This will help us page items to perform multiple requests.
  • value.contentUrl: This is the actual URL of the image. We will use this URL to download the images.

Paging Through Results

For a single API call we may get around 30 items or so by default. How do we get more images with the API? We page through the results. And the way to do this is to use the nextOffset item in the API response. We can use this value to pass in another query parameter offset to give the next page of results.

So if I only want at most 200 images, I can use the below code to page through the API results.

new_offset = 0

while new_offset <= 200:
    print(new_offset)
    params["offset"] = new_offset

    response = requests.get(url, headers=headers, params=params)
    response.raise_for_status()

    result = response.json()

    time.sleep(1)

    new_offset = result["nextOffset"]

    for item in result["value"]:
        contentUrls.append(item["contentUrl"])

We initialize the offset to 0 so the initial call will give the first page of results. In the while loop we limit to just 200 images for the offset. Within the loop we set the offset parameter to the current offset, which will be 0 initially. Then we make the API call, we sleep or wait for one second, and we set the offset parameter to the nextOffset from the results and save the contentUrl items from the results into a list. Then, we do it again until we reach the limit of our offset.

Downloading the Images

In the previous API calls all we did was capture the contentUrl items from each of the images. In order to get the images as training data we need to download them. Before we do that, let's set up our paths to be ready for images to be downloaded to them. First we set the path and then we use the os module to check if the path exists. If it doesn't, we'll create it.

dir_path = "./aston-martin/train/"

if not os.path.exists(dir_path):
    os.makedirs(dir_path)

Generally, we could just do the below code and loop through all of the content URL items and for each one we create the path with the os.path.join method to get the correct path for the system we're on, and open the path with the open method. With that we can use requests again with the get method and pass in the URL. Then, with the open function, we can write to the path from the image contents.

for url in contentUrls:
    path = os.path.join(dir_path, url)

    try:
        with open(path, "wb") as f:
            image_data = requests.get(url)

            f.write(image_data.content)
    except OSError:
        pass

However, this is a bit more complicated than we would hope it would be.

Cleaning the Image Data

If we print the image URLs for all that we get back it would look something like this:

https://www.publicdomainpictures.net/pictures/380000/velka/aston-martin-car-1609287727yik.jpg
https://images.pexels.com/photos/592253/pexels-photo-592253.jpeg?auto=compress&amp;cs=tinysrgb&amp;h=750&amp;w=1260
https://images.pexels.com/photos/2811239/pexels-photo-2811239.jpeg?cs=srgb&amp;dl=pexels-tadas-lisauskas-2811239.jpg&amp;fm=jpg
https://get.pxhere.com/photo/car-vehicle-classic-car-sports-car-vintage-car-coupe-antique-car-land-vehicle-automotive-design-austin-healey-3000-aston-martin-db2-austin-healey-100-69398.jpg
https://get.pxhere.com/photo/car-automobile-vehicle-automotive-sports-car-supercar-luxury-expensive-coupe-v8-martin-vantage-aston-land-vehicle-automotive-design-luxury-vehicle-performance-car-aston-martin-dbs-aston-martin-db9-aston-martin-virage-aston-martin-v8-aston-martin-dbs-v12-aston-martin-vantage-aston-martin-v8-vantage-2005-aston-martin-rapide-865679.jpg
https://c.pxhere.com/photos/5d/f2/car_desert_ferrari_lamborghini-1277324.jpg!d

Do you notice anything in the URLs? While most of then end in jpeg there are a few with some extra parameters on the end. If we try to download with those URLs we won't get the image. So we need to do a little bit of data cleaning here.

Luckily, there are two patterns we can check, if there is a ? in the URL and if there is a ! in the URL. With those patterns we can update our loop to download the images to the below to get the correct URLs for all images.

for url in contentUrls:
    split = url.split("/")

    last_item = split[-1]

    second_split = last_item.split("?")

    if len(second_split) > 1:
        last_item = second_split[0]

    third_split = last_item.split("!")

    if len(third_split) > 1:
        last_item = third_split[0]

    print(last_item)
    path = os.path.join(dir_path, last_item)

    try:
        with open(path, "wb") as f:
            image_data = requests.get(url)
            #image_data.raise_for_status()

            f.write(image_data.content)
    except OSError:
        pass

With this cleaning of the URLs we can get the full images.

Conclusion

While this probably isn't as sophisticated as the wrapper that FastAI has, this should help if you need to get training images from Bing Image Search manually. You can also tweak this if needed.

Using Bing Image Search is a great way to get quality and license appropriate images for training data.

An Overview of Microsoft Cognitive Services

A big thing around apps these days are that they are much more intelligent than they used to be. And with that users are expecting more and more from apps. Usually, the way to do this is to create machine learning or deep learning models yourself and deploy that within your applications. But now, Microsoft can do the heavy lifting for you with their suite of Cognitive Services.

These services can do everything you need from sentiment analysis on text to face and emotion detection. In this post, we'll go over each service that is provided so you can see what all is available and decide which one is best for your application.

Cognitive Services are divided up into several areas:

There's also a separate experimental area, called Cognitive Service Labs, where Microsoft puts newer services out for people to try and give feedback on so they can improve on them. Let's take a deeper dive into each of these services and what they offer.

Vision

Computer Vision

Perhaps one of the first of the cognitive services, the Computer Vision API does quite a lot. It can give you the following:

  • Text description of what it thinks is happening in a photo
  • Tags of what it thinks it finds in a photo
  • Content moderator scores in terms of adult content
  • How it best things to categorize the photo
  • And if it recognizes any faces within the photo such as any celebrities

That's a lot for one API to return and some of these things, such as the faces and content moderator, are their own API that you can implement separately.

Content Moderator

Do you have a site where people can upload images and videos but don't want to go through each upload to make sure there's no adult content? This API will do all of that for you.

The API can look at images to detect if it has a high confidence that it is adult content, but it can also detect if the image is racy. That is, if it is midly sexual but not considered adult. It's no limited to just images, though. It can also do these checks with video.

This can also look at text to see if there is any profanity in it. But it goes beyond just profanity. It can also help check if there is any personal identifiable information (PII) in the text to make sure no personal information is published to your site. In these times of digital privacy being even more important, this is going to be very helpful.

Face API

The Face API lets you do a few things in terms of finding and identifying people based on their face in a photo. This includes verifying faces from two photos match, recognizing emotion from a person's face, and detecting faces in a photo.

The emotion detection is interesting in that the API response will give you scores and the highest scored emotion is the one it thinks the face is giving.

"scores": {
      "anger": 0.09557262,
      "contempt": 0.003917685,
      "disgust": 0.684764564,
      "fear": 4.03712329E-06,
      "happiness": 8.999826E-08,
      "neutral": 0.002147009,
      "sadness": 0.213587672,
      "surprise": 6.34691469E-06
    }

In this response, "disgust" was the highest with a score of almost 70%.

Video Indexer

The Video Indexer API is really interesting. It combines a few of the other APIs including content moderation, speech, and sentiment analysis to give a complete look at a video. For a complete walkthrough at what it can do, I have a Wintellect post that details the video indexer.

Custom Vision

The Customer Vision API is where you get to upload your own sets of images, tag them, and then train a model to classify those images into the tags. This is a lot more powerful than just using the regular Computer Vision API where you can use your own images to train a model. This gives you or your business a more specialized model that is much easier to train and deploy than doing it by hand.

Speech

Speech to Text and Text to Speech

These are similar and are mostly self explanatory, but there are some subtlties that Microsoft gives you with them. For instance, the APIs can be customized to detect specific vocabularies and accents for a more accurate detection of the speech.

Speaker Recognition

This API can recognize who is speaking of an unknown speaker and gives you the speaker's name. This works by sending in different recordings of a person's voice and associating them with that voice. Then the API knows how to recognize their voice and it works quite well.

This API can also do speaker verification. Similar to enrolling the voice for recognition, you'd have to train the API to know who is speaking so it can recognize the voice.

Speech Translation

A fairly simple API but a powerful one. This adds real time and multi-language translation to speech in your apps. So if your application needs to support this type of feature, then just implement this API instead of spending the resources to create this model on your own, which may not even be as accurate as Microsoft's.

Knowledge

QnA Maker

This API is unique and serves to solve a specific problem - can automate responses to commonly asked questions about my site or services? QnA Maker makes this really easy by having a web interface where you can put the questions and their answers in and then train a model to detect the question being asked and output the correct answer to it.

This API is mainly used in conjunction with other services, specifically LUIS, that will be highlighted in the language section, and the Bot Framework to create bot to answer those types of questions for your users.

Custom Decision Service

This API is a newer offering and is still in preview, but is also one to look out for. It is a way to offer custom recommendations on certain items. Not only that, but it does this in real time through reinforcement learning.

This simple yet powerful API gives your applications the ability to search the web through a keyword. The API brings back search results of web pages, images, video, and even news with a single API call.

This API allows you to search with an image and it will return any results that match that image. For example, if a customer wants to purchase a similar couch to replace their current one, your application can give results based off of a photo of their couch that they take and give product recommendations based on the results.

This API can give some extra context about people, places, and things that your application may refer to based off what is available out on the web about them. As an example, suppose your application gives recommendations on local places to visit depending on where your user is. With this API you can give more context about the items that your application finds to your user so they can make a more informed decision on where to go or what to do next.

This video search API does the same thing as the web search API, but it only returns videos. However, this API gives a lot more in the results than just links to videos; you also get information such as the embed HTML, view count, and video length for starters. This is a perfect API if your application content is all about videos from around the web.

Another specific API, this one returns news articles. Similar to the Video Search API, this one returns a lot of additional information you can use within your application. For instance, along with the news article URL, you get the date it was created, related news and categories to the current news article, and specific mentions within the article.

You can't have specific search on news and videos without having one that's for images. Not only does this API give you the full URL to the image, but you also get metadata on each image, thumbnails, and information on the website that published the image.

While the other search APIs are for the entire web, the Custom Search API will allow you to have search capabilities on your own custom domain. This means that search results will only be relevant to your domain and nothing else unless you specify other web pages to include. An example for this API can be if you have a huge knowledge base of FAQ on your site about all the products your company uses, then incorporating the Custom Search API will let users search through the entire knowledge base so they can quickly find an answer instead of browsing through all of the content.

Bing Autosuggest

This is a unique API in all it does is, as you type in a search box, the API gets called giving suggestions on what it thinks you're typing. You've seen this if you've done a Google search, but to help illustrate, and see some results this API gives, here's a gif:

autosuggest.gif

Language

Text Analytics

This API is probably the most essential when you want to get information from text. That's due to what all this API gives you. You get sentiment analysis, it detects what language the text is in, and it extracts key phrases.

The API has actually been updated to also include a way to identify entities within the text. What this gives you is a way to tell if the text refers to a specific place, person, or organization.

2018-05-25 09_06_54-Clipboard.png

Bing Spell Check

Another interesting API offering, the Spell Check API will analyze your text and give any suggestions on what it thinks should be fixed.

For instance, if I give the API the text of He has fixed there enviornment, it will come back with the following JSON:

{
  "_type": "SpellCheck",
  "flaggedTokens": [
    {
      "offset": 13,
      "token": "there",
      "type": "UnknownToken",
      "suggestions": [
        {
          "suggestion": "their",
          "score": 0.786016486722212
        },
        {
          "suggestion": "the",
          "score": 0.702565790712439
        }
      ]
    },
    {
      "offset": 19,
      "token": "enviornment",
      "type": "UnknownToken",
      "suggestions": [
        {
          "suggestion": "environment",
          "score": 0.786016486722212
        }
      ]
    }
  ]
}

Translator Text

This API can translate text between, as of right now, over 60 languages. Since it is REST based, there are an endless amount of applications for this API, such as helping you to localize your applications, communication and messaging applications, or you can mix in the Vision API to read text from images and translate that to another language with this API, and use the Speech API to speak the translation out from your application.

Content Moderator

This simple API looks at images and video to determine if they have adult content in it. This is a very useful API to help you make sure no adult images get uploaded to your website if you have that feature.

This API also includes a flag in the response if the image or video is racy or not. If it is racy it isn't necessarily adult, but somewhat close to it. For example, this API may flag a photo as racy if it is of a person in a bathing suit.

Language Understanding

Language Understanding Intelligent Service, or LUIS, is a nice part of the Cognitive Services family that allows applications to understand natural language. I've also talked more in detail about LUIS in a Wintellect post as well as how to integrate LUIS into the bot framework.

Cognitive Service Labs

Gesture

This API allows you to implement custom gestures into your application. And by gestures, it lets you implement hand gestures. Using hand gestures in your application can allow your users to be more productive and your applications be more intuitive to use.

Ink Analysis

Ink Analysis allows you to not just allow a user to use a digital pen within your application, but it can understand what they wrote so you can take action on it.

This API can understand several things, such as:

  • Shapes
  • Handwritten text in over 60 languages
  • Layout of text

Local Insights

This fairly simple API can give you scores on locations around you by the different types of aminities that those places have. You can also tell it to score based on the proximity to you, or by other custom criteria.

Event Tracking

This simple API will return events based off of a Wikipedia entry. For example, if I give a query of "Microsoft" I'll get a response like the below:

{
      "relatedEntities": [
        "Microsoft",
        "Surface 3"
      ],
      "latestDocument": {
        "documentId": "ceeddb607592dc8260110855489fa45b",
        "source": "Neowin",
        "publishDate": "2018-06-01T21:22:01Z",
        "title": "Surface 3 gets its first firmware update in a year and a half",
        "summary": "Microsoft today released a firmware update for its Surface 3 tablet, and it's the first update that the device has received since September 27, 2016.It contains security and display fixes.",
        "url": "https://www.neowin.net/news/surface-3-gets-its-first-firmware-update-in-a-year-and-a-half",
        "entities": [
          "Microsoft",
          "Surface 3"
        ]
      }
    },
    {
      "relatedEntities": [
        "Paul Allen",
        "Bill & Melinda Gates Foundation",
        "Government of India"
      ],
      "latestDocument": {
        "documentId": "ceeddc75d6a8971302b102291211f79f",
        "source": "UW CSE News",
        "publishDate": "2018-06-01T20:08:03Z",
        "title": "Thank you to our state legislators!",
        "summary": "On Thursday the Paul G. Allen School was honored to host UW’s annual reception thanking our state legislators for their investments in education.In the case of the Allen School, recent investments include substantial support for the Bill & Melinda Gates Center – a second building that will double our space when it opens in January – and multiple years of funding for enrollment growth that have more than doubled our degree capacity.",
        "url": "https://news.cs.washington.edu/2018/06/01/thank-you-to-our-state-legislators/",
        "entities": [
          "Paul Allen",
          "Bill & Melinda Gates Foundation",
          "Government of India"
        ]
      }
    }

This API is designed to make search in your application much more efficient and faster. It does this by recognizing entities with search queries to determine what the query is about and allows your users to get quicker answers.

Personality Chat

This will easily enable your bots to have more small talk, or conversation about unimportant subjects. Specifically, Personality Chat can add the following:

  • Remove fallback answers such as "I don't know", or "I didn't get that".
  • Add personality to the small talk.
  • Add customizable small talk responses.
Untitled Project.gif

Knowledge Exploration

This API can help enable search experiences over data with natural language queries. If you've used the Q&A feature in Power BI then you've seen an example of Knowledge Exploration.

Untitled Project.gif

If you're curious about more Power BI, feel free to checkout this post on the Wintellect blog.

Academic Knowledge

This API will allow you to get insights of academic content, mainly researach papers. Similar to Knowledge Exploration this API can interpret natural language queries to return more accurate results.

URL Preview

This API is specific in that it provides a URL's title, description, and a relevent image. This API will also let you know if the URL is flagged for adult content.

Conversation Learner

Similar to LUIS, Conversation Learner allows you to train models based on your own criteria for dialogs. Mainly to be used for the Bot Framework, this will definitely help people develop better bots to help their customers.

Anomaly Finder

This API will help you find anomalies, or outliers, in your data. Mainly used for time series data it will return back which of your data points is considered an anomaly.

Custom Decision

This API is interesting as it will allow a recommendation engine as a sevice. Using reinforcement learning, it gives personalized content based on the information that you provide to it.


It was a long journey, but we have went through each of the current offerings from Microsoft's Cognitive Services. Feel free to try any of these as they will really help to make your applications much more intelligent.