In this code pattern, we walk you through a working example of a web app that utilizes multiple Watson services to create a better customer care experience.
Using the Watson Discovery Smart Document Understanding (SDU) feature, we will enhance the Discovery model so that queries will be better focused to only search the most relevant information found in a typical owner's manual.
Using Watson Assistant, we will use a standard customer care dialog to handle a typical conversation between a customer and a company representitive. When a customer question involves operation of a product, the Assistant dialog skill will communicate with the Discovery service using an Assistant search skill.
NOTE: This code pattern includes instructions for accessing Watson services running on both IBM Cloud and on IBM Cloud Pak for Data. In setting up your app, the main difference is the additional credentials required to access the IBM Cloud Pak for Data cluster that is hosting your Watson services.
Click here for more information about IBM Cloud Pak for Data.
SDU trains Watson Discovery to extract custom fields in your documents. Customizing how your documents are indexed into Discovery will improve the answers returned from queries.
With SDU, you annotate fields within your documents to train custom conversion models. As you annotate, Watson is learning and will start predicting annotations. SDU models can also be exported and used on other collections.
Current document type support for SDU is based on your plan:
- Lite plans: PDF, Word, PowerPoint, Excel, JSON, HTML
- Advanced plans: PDF, Word, PowerPoint, Excel, PNG, TIFF, JPG, JSON, HTML
Here is a great video that provides an overview of the benefits of SDU, and a walk-through of how to apply it to your document:
An Assistant search skill is a mechanism that allows you to directly query a Watson Discovery collection from your Assistant dialog. A search skill is triggered when the dialog reaches a node that has a search skill enabled. The user query is then passed to the Watson Discovery collection via the search skill, and the results are returned to the dialog for display to the user.
Click here for more information about the Watson Assistant search skill.
NOTE: Another method of integrating Watson Assistant with Watson Discovery is through the use of a webhook, which can be created using IBM Cloud Functions. Click here to view a code pattern that uses this technique.
- The document is annotated using Watson Discovery SDU
- The user interacts with the backend server via the app UI. The frontend app UI is a chatbot that engages the user in a conversation.
- Dialog between the user and backend server is coordinated using a Watson Assistant dialog skill.
- If the user asks a product operation question, a search query is issued to the Watson Discovery service via a Watson Assistant search skill.
NOTE: This video is using
V1
versions of both Watson Discovery and Assistant. The concepts are similar inV2
, but the navigation steps may be different.
- Clone the repo
- Create Watson services
- Configure Watson Discovery
- Configure Watson Assistant
- Add Watson service credentials to environment file
- Run the application
git clone https://github.com/IBM/watson-assistant-with-search-skill
Create the following services:
- Watson Assistant
- Watson Discovery
The instructions will depend on whether you are provisioning services using IBM Cloud Pak for Data or on IBM Cloud.
Click to expand one:
IBM Cloud Pak for Data
Use the following instructions for each of the services.
The services are not available by default. An administrator must install them on the IBM Cloud Pak for Data platform, and you must be given access to the service. To determine whether the service is installed, Click the Services icon () and check whether the service is enabled.
IBM Cloud
- If you do not have an IBM Cloud account, register for a free trial account here.
- Create a Assistant instance from the catalog.
- Create a Discovery instance from the catalog and select the default "Plus" plan.
NOTE: The first instance of the
Plus
plan for IBM Watson Discovery comes with a free 30-day trial; it is chargeable once the trial is over. If you no longer require your Plus instance for Watson Discovery after going through this exercise, feel free to delete it.
Start by launching your Watson Discovery instance. How you do this will depend on whether you provisioned the instance on IBM Cloud Pak for Data or on IBM Cloud.
Click to expand one:
IBM Cloud Pak for Data
Find the Discovery service in your list of Provisioned Instances
in your IBM Cloud Pak for Data Dashboard.
Click on View Details
from the options menu associated with your Discovery service.
Click on Open Watson Discovery
.
IBM Cloud
From the IBM Cloud dashboard, click on your new Discovery service in the resource list.
From the Manage
tab panel of your Discovery service, click the Launch Watson Discovery
button.
The landing page for Watson Discovery is a panel showing your current projects.
Create a new project by clicking the New Project
tile.
NOTE: The Watson Discovery service queries are defaulted to be performed on all collections within a project. For this reason, it is advised that you create a new project to contain the collection we will be creating for this code pattern.
Give the project a unique name and select the Document Retrieval
option, then click Next
.
For data source, click on the Upload data
tile and click Next
.
Enter a unique name for your collection and click Finish
.
On the Configure Collection
panel, click the Drag and drop files here or upload
button to select and upload the ecobee3_UserGuide.pdf
file located in the data
directory of your local repo.
NOTE: The
Ecobee
is a popular residential thermostat that has a wifi interface and multiple configuration options.
Once the file is loaded, click the Finish
button.
Note that after the file is loaded it may take some time for Watson Discovery to process the file and make it available for use. You should see a notification once the file is ready.
To access the collection, make sure you are in the correct project, then click the Manage Collections
tab in the left-side of the panel.
Click the collection tile to access it.
Before applying Smart Document Understanding (SDU) to our document, lets do some simple queries on the data so that we can compare it to results found after applying SDU. Click the Try it out
panel to bring up the query panel.
Enter queries related to the operation of the thermostat and view the results. As you will see, the results are not very useful, and in some cases, not even related to the question.
Now let's apply SDU to our document to see if we can generate some better query responses.
From the Define structure
drop-down menu, click on New fields
.
From the Identify fields
tab panel, click on User-trained models
, the click Submit
from the confirmation dialog.
Click the Apply changes and reprocess
button in the top right corner. This will SDU process.
Here is the layout of the SDU annotation panel:
The goal is to annotate all of the pages in the document so Discovery can learn what text is important, and what text can be ignored.
- [1] is the list of pages in the manual. As each is processed, a green check mark will appear on the page.
- [2] is the current page being annotated.
- [3] is a graphic display of the same page, but allows you to select regions that you can label.
- [4] is the list of labels you can assign to the graphic page.
- Click [5] to submit the page to Discovery.
- Click [6] when you have completed the annotation process.
To add/change a label, select the new label, then click on the text areas in the graphic page to apply it.
As you go though the annotations one page at a time, Discovery is learning and should start automatically updating the upcoming pages. Once you get to a page that is already correctly annotated, you can stop, or simply click Submit
[5] to acknowledge it is correct. The more pages you annotate, the better the model will be trained.
For this specific owner's manual, at a minimum, it is suggested to mark the following:
- The main title page as
title
- The table of contents (shown in the first few pages) as
table_of_contents
- All headers and sub-headers (typed in light green text) as a
subtitle
- All page numbers as
footers
- All circuitry diagram pages (located near the end of the document) as a
footer
- All licensing information (located in the last few pages) as a
footer
- All other text should be marked as
text
.
Click the Apply changes and reprocess
button [6] to load your changes. Wait for Discovery to notify you that the reprocessing is complete.
Next, click on the Manage fields
[1] tab.
- [2] Here is where you tell Discovery which fields to ignore. Using the
on/off
buttons, turn off all labels exceptsubtitles
andtext
. - [3] is telling Discovery to split the document apart, based on
subtitle
. - Click [4] to submit your changes.
Return to the Activity
tab. After the changes are processed (may take some time), your collection will look very different:
Return to the query panel (click Try it out
) and see how much better the results are.
The instructions for configuring Watson Assistant are basically the same for both IBM Cloud Pak for Data and IBM Cloud.
One difference is how you launch the Watson Assistant service. Click to expand one:
IBM Cloud Pak for Data
Find the Assistant service in your list of Provisioned Instances
in your IBM Cloud Pak for Data Dashboard.
Click on View Details
from the options menu associated with your Assistant service.
Click on Open Watson Assistant
.
IBM Cloud
Find the Assistant service in your IBM Cloud Dashboard.
Click on the service and then click on Launch tool.
From the main Assistant panel, you will see 2 tab options - Assistants
and Skills
. An Assistant
is the container for a set of Skills
.
Go to the Assistant tab and click Create assistant
.
Give your assistant a unique name then click Create assistant
.
You will then be taken to a panel that shows the Skills
assigned to your Assistant
. You can also revisit this panel by selected the desired Assistant
listed in the Assistants
tab panl.
Click on Add dialog skill
.
From the Add Dialog Skill
panel, select the Use sample skill
tab.
Select the Customer Care Sample Skill
as your template.
The newly created dialog skill should now be shown in your Assistant panel:
Click on your newly created dialog skill to edit it.
The default customer care dialog does not have a way to deal with any questions involving outside resources, so we will need to add this.
Create a new intent
that can detect when the user is asking about operating the Ecobee thermostat.
From the Customer Care Sample Skill
panel, select the Intents
tab.
Click the Create intent
button.
Name the intent #Product_Information
, and at a minimum, enter the following example questions to be associated with it.
Now we need to add a node to handle our intent. Click on the back arrow in the top left corner of the panel to return to the main panel. Click on the Dialog
tab to bring up the nodes defined for the dialog.
Select the What can I do
node, then click on the drop down menu and select the Add node below
option.
Name the node "Ask about product" [1] and assign it our new intent [2].
In the Assistant responds
dropdown, select the option Search skill
.
This means that if Watson Assistant recognizes a user input such as "how do I set the time?", it will direct the conversation to this node, which will integrate with the search skill.
Return to the Skills panel by clicking the Skills
icon in the left menu pane.
From your Assistant skills panel, click on Create skill
.
For Skill Type
, select Search skill
and click Next
.
Note: If you have provisioned Watson Assistant on IBM Cloud, the search skill is only offered on a paid plan, but a 30-day trial version is available if you click on the
Plus
button.
Give your search skill a unique name, then click Continue
.
From the search skill panel, select the Discovery service instance and collection you created previously.
Click Configure
to continue.
From the Configure Search Response
panel, select text
as the field to use for the Body
of the response.
You can also customize the return Message
to be more appropriate for your use case.
Click Create
to complete the configuration.
Now when the dialog skill node invokes the search skill, the search skill will query the Discovery collection and display the text result to the user.
NOTE: The following feature is currently only available for Watson Assistant provisioned on IBM Cloud.
You should now see both skills have been added to your assistant.
Normally, you can test the dialog skill be selecting the Try it
button located at the top right side of the dialog skill panel, but when integrated with a search skill, a different method of testing must be used.
From your assistant panel, select Preview link
.
From the list of available integration types, select Preview link
.
If you click on the generated URL link, you will be able to interact with your dialog skill. Note that the input "how do I turn on the heater?" has triggered our Ask about product
dialog node and invoked our search skill.
Copy the local env.sample
file and rename it .env
:
cp env.sample .env
You will need to Update the .env
file with the credentials from your Assistant service. First you will need the Assistant ID
which can found by:
- Click on your Assistant panel
- Clicking on the three dots in the upper right-hand corner and select
Settings
. - Select the
API Details
tab.
You will also need the credentials for your Assistant service. What credentials you will need will depend on if you provisioned Watson Assistant on IBM Cloud Pak for Data or on IBM Cloud. Click to expand one:
IBM Cloud Pak for Data
- For production use, create a user to use for authentication. From the main navigation menu (☰), select Administer > Manage users and then + New user.
- From the main navigation menu (☰), select My instances.
- On the Provisioned instances tab, find your service instance, and then hover over the last column to find and click the ellipses icon. Choose View details.
- Copy the URL to use as the {SERVICE_NAME}_URL when you configure credentials.
- Optionally, copy the Bearer token to use in development testing only. It is not recommended to use the bearer token except during testing and development because that token does not expire.
- Use the Menu and select Users and + Add user to grant your user access to this service instance. This is the user name (and password) you will use when you configure credentials to allow the Node.js server to authenticate.
Edit the .env
file with the necessary credentials and settings.
# Copy this file to .env and replace the credentials with
# your own before starting the app.
#----------------------------------------------------------
# IBM Cloud Pak for Data (username and password)
#
# If your services are running on IBM Cloud Pak for Data,
# uncomment and configure these.
# Remove or comment out the IBM Cloud section.
#----------------------------------------------------------
ASSISTANT_AUTH_TYPE=cp4d
ASSISTANT_AUTH_URL=https://my-cpd-cluster.ibmcodetest.us
ASSISTANT_USERNAME=my-username
ASSISTANT_PASSWORD=my-password
ASSISTANT_URL=https://my-cpd-cluster.ibmcodetest.us/assistant/assistant/instances/1576274722862/api
# # If you use a self-signed certificate, you need to disable SSL verification.
# # This is not secure and not recommended.
## ASSISTANT_AUTH_DISABLE_SSL=true
## ASSISTANT_DISABLE_SSL=true
ASSISTANT_ID=<add_assistant_id>
IBM Cloud
For the Watson Assistant service provisioned on IBM Cloud, all of your credentials will be available on the API Details
panel.
Important: ASSISTANT_URL should end with an instance ID. If it contains
v2/assistants/<id>
, please delete this part of the URL. For example: "https://api.us-south.assistant.watson.cloud.ibm.com/instances/5db04c67/v2/assistants/a85f67e8-xxx/sessions" should be truncated to "https://api.us-south.assistant.watson.cloud.ibm.com/instances/5db04c67".
Edit the .env
file with the necessary credentials and settings.
# Copy this file to .env and replace the credentials with
# your own before starting the app.
#----------------------------------------------------------
# IBM Cloud
#
# If your services are running on IBM Cloud,
# uncomment and configure these.
# Remove or comment out the IBM Cloud Pak for Data sections.
#----------------------------------------------------------
# Watson Assistant
ASSISTANT_AUTH_TYPE=iam
ASSISTANT_APIKEY=2ZzwqVghEhvLEvRxfxxxxxmLukVv3YIF411GvgXhHX
ASSISTANT_URL=https://api.us-south.assistant.watson.cloud.ibm.com/instances/5db04c67-b295-9999-bb99-e5587b6bed91
ASSISTANT_ID=25f892c4-e1e6-2222-b6c8-0b8660786d1f
- Install Node.js runtime or NPM.
- Start the app by running
npm install
, followed bynpm start
. - Use the chatbot at
localhost:3000
.
Sample questions:
- How do I override the scheduled temperature
- How do I turn on the heater
- how do I set a schedule?
- To view the raw JSON results returned from Discovery (via the Assistant search skill), open up the Development tools in your browser and view the console output. Here is sample of what you should see:
{
"result": {
"output": {
"generic": [
{
"response_type": "search",
"header": "I searched my knowledge base and found this information which might be useful:",
"results": [
{
"title": null,
"body": "You can override the scheduled temperature by moving the bubble on the temperature slider up or down. The blue number represents the cool set point; the orange number represents the heat set point. The new desired temperature will be the set point used for the Hold. The duration of the Hold is the last configured value (the default value is Until I change it, meaning it keeps the value indefinitely, until you choose to revert to the schedule or change it). You can adjust the default Hold time in the Preferences menu (page 21). To cancel the current Hold, touch the Hold message box displayed on the Home screen. You can touch the box anywhere and not just the X displayed on the box.",
"url": null,
"id": "3a5efee70d8cc9d70e2b94d22c15e2d1_24",
"result_metadata": {
"confidence": 0.3056213337457547,
"score": 7.38697695
},
"highlight": {
"text": [
"You can override the <em>scheduled</em> temperature by moving the bubble on the temperature slider up or down. The blue number represents the cool <em>set</em> point; the orange number represents the heat <em>set</em> point. The new desired temperature will be the <em>set</em> point used for the Hold.",
"The duration of the Hold is the last configured value (the default value is Until <em>I</em> change it, meaning it keeps the value indefinitely, until you choose to revert to the <em>schedule</em> or change it). You can adjust the default Hold time in the Preferences menu (page 21). To cancel the current Hold, touch the Hold message box displayed on the Home screen."
],
"body": [
"You can override the <em>scheduled</em> temperature by moving the bubble on the temperature slider up or down. The blue number represents the cool <em>set</em> point; the orange number represents the heat <em>set</em> point. The new desired temperature will be the <em>set</em> point used for the Hold.",
"The duration of the Hold is the last configured value (the default value is Until <em>I</em> change it, meaning it keeps the value indefinitely, until you choose to revert to the <em>schedule</em> or change it). You can adjust the default Hold time in the Preferences menu (page 21). To cancel the current Hold, touch the Hold message box displayed on the Home screen."
]
}
},
- Artificial Intelligence Code Patterns: Enjoyed this Code Pattern? Check out our other AI Code Patterns
- AI and Data Code Pattern Playlist: Bookmark our playlist with all of our Code Pattern videos
- With Watson: Want to take your Watson app to the next level? Looking to utilize Watson Brand assets? Join the With Watson program to leverage exclusive brand, marketing, and tech resources to amplify and accelerate your Watson embedded commercial solution.
This code pattern is licensed under the Apache License, Version 2. Separate third-party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 and the Apache License, Version 2.