This Code Pattern is part of the series Extracting Textual Insights from Videos with IBM Watson. Please complete the Extract audio from video, Build custom Speech to Text model with speaker diarization capabilities and Use advanced NLP and tone analysis to extract meaningful insights code patterns of the series before continuing further since all the code patterns are linked.
In a virtually connected world, staying focused towards work or education is very important. Studies suggests that most people tend to lose their focus from live virtual meetings or virtual classroom sessions post 20min, hence most of the meetings and virtual classrooms are recorded so that an individual can go through it later.
What if these recordings could be analyzed with the help of AI and a detailed report of the meeting or classroom could be generated? Towards this goal, in this code pattern, given a video recording of the virtual meeting or a virtual classroom, we will be extracting audio from video file using open source library FFMPEG, transcribing the audio to get speaker diarized notes with custom trained language and acoustic speech to text models, and generating a NLU report that consists of Category, Concepts, Emotion, Entities, Keywords, Sentiment, Top Positive Sentences and Word Clouds using Python Flask runtime.
In this code pattern, given any video, we will learn how to extract speaker diarized notes and meaningful insights report using Speech To Text, advanced NLP and Tone Analysis.
When you have completed this code pattern, you will understand how to:
- Use Watson Speech to Text service to convert the human voice into the written word.
- Use advanced NLP to analyze text and extract meta-data from content such as concepts, entities, keywords, categories, sentiment and emotion.
- Leverage Tone Analyzer's cognitive linguistic analysis to identify a variety of tones at both the sentence and document level.
-
User uploads recorded video file of the virtual meeting or a virtual classroom in the application.
-
FFMPG Library extracts audio from the video file.
-
Watson Speech To Text transcribes the audio to give a diarized textual output.
-
Watson Language Translator (Optionally) translates other languages into English transcript.
-
Watson Tone Analyzer analyses the transcript and picks up top positive statements form the transcript.
-
Watson Natural Language Understanding reads the transcript to identify key pointers from the transcript and get the sentiments and emotions.
-
The key pointers and summary of the video is then presented to the user in the application.
-
The user can then download the textual insights.
Clone the extract-textual-insights-from-video
repo locally. In a terminal, run:
$ git clone https://github.com/IBM/extract-textual-insights-from-video
You will have to add Watson Speech-To-Text, Tone Analyzer and Natural Language Understanding Credentials to the Application.
If you have completed the first three code patterns of the series, then you can add the same credentials created in second code pattern of the series and third code pattern of the series by following the steps below.
Add existing credentials created from the series
-
In the second code pattern of the series cloned repo, you will have updated speechtotext.json file with speech to text credentials. Copy both the files and paste it in parent folder of the repo that you cloned in step 1.
-
In the third code pattern of the series cloned rep, you will have updated naturallanguageunderstanding.json file with natural language understanding credentials and toneanalyzer.json file with tone analyzer credentials. Copy that file and paste it in parent folder of the repo that you cloned in step 1.
Or if you have landed on this code pattern directly without completing the previous code patterns of the series, you can add new credentials by following the steps below.
Add new credentials
- On IBM Cloud, create a Watson Speech To Text Service, and click on
create
as shown.
- In Speech To Text Dashboard, Click on
Services Credentials
- Click on
New credential
and add a service credential as shown.
-
Once the credential is created, you can copy the credentials using the small two overlapping squares and paste the credentials into speechtotext.json file present in the cloned repo.
-
Back to IBM Cloud, create a Natural Language Understanding service, under
Select a pricing
plan selectLite
and click oncreate
as shown.
- Click on New credential and add a service credential as shown.
-
Once the credential is created, you can copy the credentials using the small two overlapping squares and paste the credentials into naturallanguageunderstanding.json file present in the cloned repo.
-
Back to IBM Cloud, create a Tone Analyzer service, under
Select a pricing
plan selectLite
and click oncreate
as shown.
-
In Tone Analyzer dashboard, click on Services Credentials
-
Click on New credential and add a service credential as shown.
- Once the credential is created, you can copy the credentials using the small two overlapping squares and paste the credentials into toneanalyzer.json file present in the cloned repo.
With Docker Installed
- change directory to repo parent folder :
$ cd extract-textual-insights-from-video/
- Build the Dockerfile as follows :
$ docker image build -t extract-textual-insights-from-video .
- once the dockerfile is built run the dockerfile as follows :
$ docker run -p 8080:8080 extract-textual-insights-from-video
- The Application will be available on http://localhost:8080
Without Docker
- Install the FFMPEG library.
For Mac users run the following command:
$ brew install ffmpeg
Other platform users can refer to the ffmpeg documentation to install the library.
-
Install the python libraries as follows:
- change directory to repo parent folder
$ cd extract-textual-insights-from-video/
- use
python pip
to install the libraries
$ pip install -r requirements.txt
-
Finally run the application as follows:
$ python app.py
- The Application will be available on http://localhost:8080
- Visit http://localhost:8080 on your browser to run the application.
-
We'll begin by uploading a video from which we'll be extracting insights.
-
You can make use of any meeting video or classroom video that you have or you can download the video that we have used for the demonstration purpose.
-
This is a free educational video taken from cognitiveclasses.ai. The video is an introduction to a python course.
-
Click on the
Drag and drop files here or click here to upload
, choose the video file you want to extract insights from.
- You can see the custom language and acoustic models that we built in Build custom Speech to Text model with speaker diarization capabilities code pattern from the series. If you have landed directly on this code pattern and created credentials then you will see "Lite version".
Note: We have trained a custom language model and an acoustic model with
IBM Earnings Call Q1 2019
dataset. Hence the model's performance will be best for Computer Science, Finance related Content. The model can be trained according to the content that you wish to extract. Example: Train the model with sports dataset to get best results with sports commentary.
-
You can find the advance NLP and Tone Analyzer options that we worked with in the Use advanced NLP and Tone Analysis to extract meaningful insights code pattern from the series. If you have landed directly on this code pattern and created credentials then you will see "Lite version".
-
Click on the
Submit
button and wait for the application to process. When you have pressed submit, the application in background will:- Extract audio from the video.
- Transcribe the audio to get speaker diarized notes.
- Use advanced NLP and Tone Analysis to extract insightful report.
- As soon as the video is uploaded, you can see the video preview on the screen as shown.
-
You can track the progress through the progress bar as shown.
-
The various progressing stages are:
- Uploading
- Extracting
- Transcribing
- NLU Analysing
NOTE: An approximate time to complete the extraction of insights will be displayed.
- Once the video is transcribed you can scroll down to see the speaker diarized textual output under
Speech To Text
tab as shown.
- Similarly once the NLU Analysis is completed, you can click on the
NLU Analysis
tab to view the report.
- More about the entities:
Category
- Categorize your content using a five-level classification hierarchy. View the complete list of categories here.Concept Tags
: Identify high-level concepts that aren't necessarily directly referenced in the text.Entity
: Find people, places, events, and other types of entities mentioned in your content. View the complete list of entity types and subtypes here.Keywords
: Search your content for relevant keywords.Sentiments
: Analyze the sentiment toward specific target phrases and the sentiment of the document as a whole.Emotions
: Analyze emotion conveyed by specific target phrases or by the document as a whole.Positive sentences
: The Watson Tone Analyzer service uses linguistic analysis to detect emotional and language tones in written text
- Learn more features of:
- Watson Natural Language Understanding service. Learn more.
- Watson Tone Analyzer service. Learn more.
-
Once the NLU Analysis Report is generated you can review it. The Report consists of:
-
Features extracted by Watson Natural Language Understanding
-
Features extracted by Watson Tone Analyzer:
-
Other features
-
Category
: Based on the dataset that we used, you can see that the category was extracted astechnology and computing
specificallySoftware
.
Note : You can see the confidence score of the model in green bubble tags.
-
Entity
: As you can see entity isPerson
specificallyAlex Ackles
indicating that, in the video recording most of the emphisis is given by aPerson, Ackles
. -
Concept Tags
: Top 3 concept tags are extracted from the video,United Nations
,Aesthetics
andStatistics
indicating that the speaker spoke about these contexts more often. -
Keywords
,Sentiments
andEmotions
: Top keywords along with their sentiments and emotions are extracted, giving a sentiment analysis of the entire meeting. -
Top Positive Sentences
: Based on emotional tone and language tone, positive sentences spoken in the video is extracted and is limited to 5 top positive sentences. -
Word Clouds
: Based on the keywords,Nouns & Adjectives
as well asVerbs
are analyzed, and the result is then turned into word clouds.
- The Report can be printed by clicking on the
print
button as shown.
We learnt how to extract audio from video files, transcribe the audio with our custom built models, process the transcript to get speaker diarized notes as well as NLU analysis report.
This code pattern is licensed under the Apache License, Version 2. Separate third-party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 and the Apache License, Version 2.