Navlead

AI-powered assistant chatbot for navigation in 3D enviroments.

The chatbot was trained on real-world navigation assistance dialogs to give natural responses.

The 3D enviroments were reconstructed form real-world houses.

Overview of NavLead

Navlead is an AI-powered assistant chatbot for navigation in 3D enviroments. The simulation, the chatbot, and their deployment online are all part of this project. (Note: this repo contains the client-side section only)

The simulation has custom 3D assets, a chat dialog box, motion controls and game logic. The chatbot has a custom ML NLP model that has a seq2seq architecture and was trained on CVDN dataset. The client-host loads the simulation and mediates its interactions with the API server. The API server authenticates the request and runs the ML NLP model.

Motivation

Inspired by the rise of VR and AI, this project aims to combine both into a navigation assistance system that can run through the browser.

Tech stack of Navlead

• Client-side
  • Simulation
    • Blender (3.1)
    • Unity (2020.3 LTS)
  • UI
    • React (Next ts)
• Server-side (not on this repo)
  • API server
    • Django
  • ML service
    • Pytorch

In-depth explanation

For an in-depth explanation of this project, please, see Navlead's About page

• TLDR;
• Motivation
• 3D simulation
  • Modeling 3D enviroment in Blender
  • Adding game controls in Unity
  • Bridging simulation to browser (Unity-side)
• Client-side integration
  • Loading Unity Script
  • Bridging simulation to browser (browser-side)
  • Answer simulation question
• Server-side API
  • Request handling
• Machine Learning
  • Data
  • Training
  • Results
• References
• Authors
  • Frederick Roman
  • Homero Roman

Live website

See Navlead.

Project setup

npm install

Compiles and hot-reloads

npm run dev

Compiles and minifies for production

npm run build

Lints and fixes files

npm run lint

Acknowledgements

Vision-and-Dialog Navigation

https://arxiv.org/abs/1907.04957
https://github.com/mmurray/cvdn