Smart Classroom

SUTD UROP: Using Sensors to Enhance Learning/Invigilation in Classrooms

Table of Contents

• Usage
  • Video Conference/Call Utilities
    • Zoom
  • Face Processing
  • Dashboard
• Stakeholders
• Additional Resources

Usage

Video Conference/Call Utilities (vc_utils)

Zoom

NOTE: Look into the zoomus Python client library and consider using it once it provides support for live-streaming (check the status of this PR).

For now, we will be using Zoom as our video conference platform. Before setup, ensure that we have at least a Zoom Pro Account and login here using our account credentials.

Zoom App Setup

To create the Zoom App, go to the Zoom Marketplace and create an account-level OAuth app to access Zoom's live streaming API endpoints.

Zoom App Configuration:

• Specify the Redirect URL for OAuth to be: https://zoom.us/oauth/authorize
• App Name: Smart Classroom
• Short Description: Using sensors to enhance learning/invigilation in classrooms
• Long Description: Zoom App used to stream video data to the Python script, which would be connected to the face processing code, which would in turn pass the output data to the dashboard.
• Developer Name: James Raphael Tiovalen
• Developer Email Address: james_raphael@mymail.sutd.edu.sg
• Whitelist URL: https://zoom.us/oauth/
• Event Subscriptions:
  • Subscription Name: Meeting Live Stream
    • All users in the account
    • Specify the Event notification endpoint URL
    • Event types:
      • Start Meeting (start_meeting)
• Scopes:
  • meeting:read:admin: To get notification of a meeting being started.
  • meeting:write:admin: To get video data stream to be used by the Python app.

After which, since this is an OAuth app, we would need to install the app manually on our Zoom account for the very first time (due to security reasons) by logging in into our Zoom Marketplace account and then going to https://marketplace.zoom.us/develop/apps/<app-id>/activation to click the Install button. Click the Authorize button when prompted.

Zoom Meeting Setup (Admin)

We will be using Zoom Meetings for classroom lessons (instead of Webinars).

Follow the instructions presented here to enable and set up custom live-streaming service for the corresponding Zoom Meetings.

Go to Account Management > Account Settings > In Meeting (Advanced) > Allow live streaming the meetings, enable the setting's toggle and enable the Custom Live Streaming Service checkbox.

Zoom Meeting Setup (Host)

As the host, start the meeting and ensure that the meeting is up before the PATCH requests are being sent to Zoom's server (as detailed in the next section's steps). If everything went well after the PATCH requests are sent, a LIVE on Custom Live Streaming Service (or LIVE on inc) indicator should be shown on the top-left section of our Zoom Meeting's window/screen (just beside the green shield with checkmark logo of Meeting Information).

Zoom Script

We are using Ubuntu 20.04.1 as our Operating System. We can use a private VPS set up on DigitalOcean (as a Droplet) as our server.

Before running anything, set up a server (we are using NGINX with the RTMP protocol) and install the dependencies:

sudo apt update
sudo apt upgrade
sudo apt install -y nginx libnginx-mod-rtmp git build-essential ffmpeg libpcre3 libpcre3-dev libssl-dev zlib1g-dev
pip3 install -r requirements.txt

Remember to configure our nginx.conf NGINX configuration file:

sudo nano /etc/nginx/nginx.conf

Add these lines to the end of the nginx.conf file (with the appropriate <stream-key> value, which is livestream in our case):

rtmp {
        server {
                listen 1935;
                chunk_size 4096;
                notify_method get;

                application live {
                        live on;
                        record off;
                        push rtmp://127.0.0.1:1935/live/<stream-key>
                }
        }
}

And then, restart NGINX:

sudo systemctl restart nginx

We would need to periodically obtain the OAuth Access Token by using the provided Client ID and Client Secret to the Authorize URL and provide a valid Access Token URL. For OAuth, since we would need JavaScript to be enabled, we would need to use Selenium and acquire an accompanying webdriver (such as geckodriver or chromedriver). To do that, run:

chmod +x get_driver.sh
./get_driver.sh

To run the ZOOM VC script to prepare the Zoom App and send the start signal to initiate a livestream:

python3 zoom.py

Login to our Zoom account on the popped-up browser that appeared. Do take note that access tokens expire after 1 hour.

If signing in using a Google account, ensure that less secure app access is allowed here. If access into the Google account is still not allowed, try opening up a new window and log into this webpage first before going back to the Zoom website to try logging in using the specified Google account again. The client_id specified in the URL is of Google's OAuth 2.0 Playground service.

Improvements to the Zoom script could be made to take attendance as well (take inspiration from here and refer to this as the Zoom API endpoint implementation).

Zoom Live Stream Data

To start processing the video stream data:

python3 process.py

Face Processing

Process face data using machine learning (PyTorch) for emotion detection, gaze tracking, head pose estimation and attention analysis.

Dashboard

Display analysis output data on an elegant and beautiful dashboard to provide real-time feedback to the instructors during class.

Stakeholders

• Team Members:
  • James Raphael Tiovalen
  • Velusamy Sathiakumar Ragul Balaji
  • Kenneth Chin Choon Hean
• Supervisor: Prof. Kenny Choo

Additional Resources

More content coming soon!