Stream and Upload Video Powered by Proven Celery Pipeline, S3, Gcore CDN and PlayReady DRM Watch in Action

CuteTube is a Video On Demand (VoD) Platform Just Like YouTube

Read the blog »

General Information | CuteTube Monolith Version

CuteTube is a High Performance video streaming platform just like YouTube. A viewer can signup in CuteTube and watch their favorite videos as well as upload their video for others to watch.

CuteTube is primarily tries to mimic the core backend of any established VoD platforms. The backend of CuteTube is built on Django.

The CuteTube platform is backed by the following core technologies -

a. Django - The Backend
b. Proven Celery Pipeline - For Asynchronous Video Processing  
c. AWS S3 - Store the DASH Segments
d. FFmpeg - Transcode, Segment Videos in MPEG-DASH with Adaptive Bitrate Streaming Technology
e. Gcore CDN - Distribute the Content Globally with Low Latency
g. Shaka Packager - Segment and Encrypt the Video with AES 128-bit Key
h. PlayReady DRM (Test Server) - Video is Protected with Digital Rights Management by PlayReady Test Licence Server  

NOTE

A. Architectures of CuteTube | Monolith and Microservices

The current implementation you are now reading is built in Monolith architecture.

There is another version of CuteTube as Microserivices where I have separated services based on concern. The microservices version of CuteTube is more robust and scalable solution with wider range of technologies implemented.

Please see the CuteTube microservices version here - CuteTube Microservices

B. Versioning of CuteTube

The CuteTube project has 4 versions. Versioning is simply the different types of implementation of CuteTube. Hence, each version (see in core_apps.stream_v*) is built differently incorporating basic to advanced technologies. The technology and the range of technology adoption increases as the version increases.

Version 01 =>

The basic implementation of VoD kind streaming where the backend itself takes a video and returns the video as chunk to the viewer.

Version 02 =>

The second version is involved with ffmpeg in CuteTube. The video is segmented and the backend serves the .mpd file and the segments to the user.

Version 03 =>

The third version is a robust and well tested solution for a VoD backend. All the necessary APIs in this version are fully automated. Celery Pipeline, S3 Bucket Storage, Gcore CDN for global distribution is implemented in this version.

Version 04 =>

In the version four, everything that is available in version 03 is available. But the addition is that DRM (Digital Rights Management) is implemented in this version with Microsoft's PlayReady DRM service. A DRM helps to encrypt the video segments and the client is required to obtain a licence from the DRM Licence Server in order to decrypt the video segment to watch. Shaka Packager has been used for segmentation purpose in this version of CuteTube. However, version 04 is still under development as the free Microsoft PlayReady test DRM licence server does not work as expected all the time and as it is nearly impossible to get free/trial DRM servers for individual other than PlayReady test DRM licence server.

Additional Note =>

CuteTube is fully dockerized solution.

C. Rate Limit Alert

Note that the project is not deployed as the celery pipeline takes too much of resources and a free server can't handle it!!

The API endpoint to for stream and upload video in the CuteTube platform is Rate Limited.

    GET https://cutetube.algocode.site/api/v3/vod/metadata/stream/<str:video_id>/

Fetch Video Metadata to stream video API is rate limited to 1 requests per 20 seconds.

    POST https://cutetube.algocode.site/api/v3/vod/initiate-task/upload/

Upload video to CuteTube API is rate limited to 1 request per 20 minutes

You can learn more on API Documentation in the API Guideline - CuteTube APIs section.

D. Containerized Solution

The project is fully containerized using docker. A make docker-up would run the full system locally provided the .envs.

E. Documentation

Please visit the documentation page localhost:8080/doc/ for more information related to API documentation.

PS - Read Deployment section to learn why localhost.

Deployment

Deployment Information

Initially I planned to deploy CuteTube Backend on AWS EC2 in Ubuntu 22.04 server. I have a free AWS server but it is impossible to deploy the project in real server as the Celery Pipeline needs too much computing power that a free AWS server can not provide. The Celery Pipeline itself need at least 3 GB of RAM to continue processing video, where as a free AWS server only provides an EC2 instance with 1 GB of RAM only!

However, I have already attached a detailed video in the Watch In Action section how CuteTube works in the background.

Features

Features of CuteTube

Small Note

As of today I have built the backend platform, and there's no frontend for the project. I am fully focusing on the advanced backend engineering, hence, if you want to contribute or want to build a frontend for the project, please do not hesitate to email me here:

A. Authentication

• The authentication system of CuteTube is built from scratch. No 3rd party packages has been used.

B. Stream

• Users can request to watch video with video_id.

• The backend service returns the metadata of the video and the dash player plays the video based on the OS of the client.

• The video is served from Gcore CDN and AWS S3 as the origin of the CDN.

• The CDN is configured with a custom domain cdn.algocode.site to serve segments to the client.

• The video segments are available for mp4 and mov container.

C. Upload

• Any authenticated user can upload video to CuteTube platform to let watch other users.

• The video processing is overloaded to a celery pipeline for asynchronous processing, and the user gets an immediate response with process metadata.

• To learn more on the workflow, please take a look at CuteTube - Architecture and Workflow Section.

Watch In Action

CuteTube Monolith In Action

• Watch on YouTube

Timeline:

1. Introduction: 00:00

2. HLD of CuteTube: 09:00

3. Upload Video to CuteTube: 24:30

4. Work Distribution on Celery Pipeline: 37:00

5. S3 Storage of DASH Segments: 46:00

6. DASH Player: 48:00

7. Watch the Uploaded Video: 50:00

8. Rate Limit: 52:15

9. Flower for Celery Monitoring: 54:15

Architecture and Workflow | HLD

Architecture of CuteTube (Monolith Version)

Workflow of CuteTube (Monolith Version)

A. Authentication

• The user can stream i.e. watch video in CuteTube without being authenticated.

• The user needs to be authenticated to upload video in CuteTube.

B. Stream Video

• No authentication is needed to watch video in CuteTube.

• User requests with video_id to fetch the video metadata.

• The video metadata is returned to the user

• The DASH Segments type are decided based on OS of the client. If the OS is Windows, the MP4 DASH segments are played, and for MacOS and Linux, MOV DASH Segments are played.

• The video is played with ABR (Adaptive Bitrate Technology). The DASH player automatically upgrades or downgrades to the appropriate bitrate based on the network condition of the client.

• The ABR supports 360, 480, 720 and 1080 pixels at 800, 1200, 2400 and 4800 Kbps respectively.

• The Dash segments are served through Gcore CDN and S3 as the upstream of the CDN.

• The CDN domain is cdn.alogcode.site.

C. Upload Video

• Authentication is needed to upload video to CuteTube.

• The user sends the video file and the video metadata such as video name, description etc. through an API.

• The backend service saves the video locally, initiates a celery pipeline and immediately responses to the client with process and video_id.

• The celery pipeline does the following when initiated :-

  • Tracks the original video format i.e. mov or mp4. CuteTube currently processes video with mov and mp4.

  • Transcodes the video to mov container if the original video is in mp4 container and vice-versa.

  • Creates a few celery group, chord and callbacks, to further process the both videos: Multiple Celery processes and tasks are responsible for below workflow : -

    • The videos are segments with ABR technology.

    • The segments are prepared in group of batches to initiate upload in S3.

    • The segmet batches are uploaded to S3 processing the batches.

    • The local files are deleted from the local storage and update metadata if needed as callback.

  • The failed tasks are retried with exponential backoff method not to overwhelm the server.

• However, the Microservices version of CuteTube triggers message queue events for producer at this stage to update the state of the process and to send notification or email to the user as a token of completion of the video upload process.

• Please see the Microservices Version of CuteTube to learn more.

API Guideline - Registration in CuteTube

Registration in the CuteTube

Why localhost? You probably already know the reason from the Deployment section!

    POST https://localhost:8080/api/v3/auth/signup/

Parameter	Type	Description
username	string	Required Your username for the account.
email	string	Required Your valid email address.
password	string	Required Your password.
password2	string	Required Confirm your password.
first_name	string	Required Your first name.
last_name	string	Required Your last name.

Login in CuteTube

    POST https://localhost:8080/api/v3/auth/login/

Parameter	Type	Description
credential	string	Required Your registered email address or your username.
password	string	Required Your password.

API Guideline - Stream Video on CuteTube

Stream Video on CuteTube

Why localhost? You probably already know the reason from the Deployment section!

    GET https://localhost:8080/api/v3/vod/metadata/stream/<video_id>/

Parameter	Type	Description
video_id	string	Required Video ID of the video user wants to stream.

API Guideline - Upload Video on CuteTube

Upload Video on CuteTube

Why localhost? You probably already know the reason from the Deployment section!

    POST https://localhost:8080/api/v3/vod/initiate-task/upload/

Parameter	Type	Description
title	string	Required The title of the Video.
description	string	Required A description of the video.
duration	string	Required Duration of the video in HH:MM:SS format.
video	file	Required A video file either in .mov or in .mp4 format.

Contributing and Run Locally

Contribution and Development

If you want to contribute or you want to run locally, then you can fork the development branch on each service mentioned in the CuteTube Platform.

Please follow the .envs-examples to know the env-variables you would need to run the project locally.

All the services are dockerized project. You just need to cd src, create virtual environment, activate it, and run make docker-up and That's it!

PS: make will only work if you're using a linux or MacOS machine and subject to install makefile in your system.

Otherwise, you may need to copy the command from the Makefile and run the commands.

This will run the project for you.

Please follow the service that you want to contribute or run locally to get detailed guideline on local development.

Lessons Learnt and Challenges

The Backstage

The project itself was a challenge for me!

Once one of my mentors told me

Do the hard things while you are learning, so that the implementation becomes easier for you.

I completely agree with this statement. I enjoy dealing with complex stuff, and bugs give me the kick I enjoy!

Well, enough praise of myself.

And I am writing this Readme today that I have completed the project, and somehow I have made it! That's my motivation.

I know something is not simple as it sounds, but I know, somehow I would manage it!

Challenges

• The initial challenge was the design. Designing a complex project like Video On Demand in microservices to build from scratch was not easy as it sounds.

• The communication between microservices were fun discovery. I was searching for optimal solution and I learnt RabbitMQ for this cause, and I ended up writing a nice blog on RabbitMQ 101. Read RabbitMQ 101 Here. The Algocode platform is using an RabbitMQ instance from CloudAMQP platform.

• I had to re-learn almost everything related to streaming industry. I have to read intense amount of research on ffmpeg. I knew nothing about transcoding, segmentation, Adaptive Bitrate Streaming, DASH, HLS, Celery Pipeline but building CuteTube taught me a lot on these technologies.

  • However, do you know I have also built a low latency live video streaming platform just like Twitch named as ProStream? Please checkout ProStream here. You'd love to interact with the project, I promise!

• The most difficult domain was to build the Celery Pipeline as it is the core structure for asynchronous processing. I have spent countless sleepless nights just to align the Celery Pipeline is well suited and well tested for various use-cases.

• The microservices version of CuteTube was more complex, I had to learn more about message queue event management to properly Acknowledge tasks based on factors deciding whether the child or related task was fullfiled or not.

Learnings

• I have gained practical experience with RabbitMQ building this project.

• I have gained deep knowledge on docker, docker volumes, docker networking, Lunux internals and many more.

• As the project is heavily dealing with files, I have gained valuable experience with file handling with python.

• As I have built the project from research, design, dev, production to deployment, I have gained invaluable knowledge on design, development, production and deploy the project in cloud services like AWS or Azure.

• As the project also built in microservises architecture, I have gained practical knowledge on communication, networking, between all other services; experience with cloud providers such a AWS, Azure and onverall dev to production of a SDLC.

• As I have built the project in both - monolith and microservices architecture, I have gained practical experience, advantages, disadvantages on certain parameters for a project on both architectures. I can make more critical judgement on system design how a certain service would behave on monolith and microservices to maximize the SDLC process.

🔗 Links