Current videoconferencing tools are nonsense: they make you send gigabytes of HD video that contain just a few relevant information. What if, instead of directly sending the video from your webcam, you were only sending your face expression and your lips movements? Then, we could render this on 3D avatars that we would place in a virtual meeting room.
This repository hosts the code of a proof of concept of the above idea, which has been submitted to COVID-19 Global Hackathon.
Features
- Create and join a room
- Speak with your peers (audio channel)
- Share your screen with your peers on the TV
- View people through a 3D avatar
- Real-time face detection and mouth opening coefficient computation (using webcam stream)
Not implemented yet
- Share FaceFilter result (face motion and mouth opening coefficient) to peers through WebRTC data channel
- Render motion on avatars head, and render lip-sync (could be computed at runtime using aframe-extras).
Ideas
- Users should be assigned a seat in a deterministic way, so everybody see the same result.
- Unicode not working in
<a-text>(see this). Need to find an alternative (generate custom font, or render an image). - Design more rooms with different capabilities/ambiances (auditorium, classroom, lounge, ...), and give the choice to the user.
- Add more characters, and give the choice to the user. Might use meshConverter, FBX2glTF or COLLADA2GLTF.
- Detect the current speaker, add an arrow on top of it.
- Ability to see shared screen in full screen, or open it in a resizable popup.
- Voice sound level could depend on distance with other people.
- WebGL consumes lot of resources in the browser. Maybe we could leverage WebWorkers for face tracking inference?
- Only display character faces in mobile mode to improve usability and performances.
While we have a bunch of good ideas to make Avatarz a really cool tool, we don't have the required skills in 3D to render expressions on avatars. We would love to get contributions from anyone willing to help!
- A New Kind of Social Network: Emotional Intelligence
- Facebook Social VR Demo - Oculus Connect 2016
- Wired - Second Life Interviews Mean Real Life Jobs
- NPR - Virtual Recruiting for Real-World Jobs
- Facebook is building the future of connection with lifelike avatars