[docs] active learning + video player (#64)

docs/index.rst

@@ -6,7 +6,8 @@
 Lightning Pose App documentation
 ================================
 
-The **Lightning Pose App** is a browser-based GUI that facilitates the deveopment of pose estimation projects.
+The **Lightning Pose App** is a browser-based GUI that facilitates the deveopment of pose
+estimation projects.
 
 The app runs in the cloud using `Lightning.ai <https://lightning.ai>`_.
 

docs/source/tabs/extract_frames.rst

@@ -12,6 +12,7 @@ The Extract Frames tab allows you to select frames for labeling using various me
 
 * :ref:`Upload videos and automatically extract random frames <upload_video_random>`
 * :ref:`Upload zipped files of frames <upload_zipped_frames>`
+* :ref:`Automatically extract frames using a given model <active_learning>`
 
 .. _upload_video_random:
 
@@ -115,3 +116,38 @@ Click "Extract frames" once the zip file upload is complete.
 Once all frames have been extracted you will see "Proceed to the next tab to label frames" in green.
 
 .. image:: https://imgur.com/F9y1aPv.png
+
+
+.. _active_learning:
+
+Automatically extract frames using a given model
+================================================
+
+.. note::
+
+    This option will not appear until at least one model has been trained.
+
+This option allows you to choose frames to label that are "difficult" for a given model.
+Since there is no ground truth, frames are selected based on likelihood values and other metrics
+that are correlated with pixel error (large temporal jumps and PCA reprojection errors;
+see the original Lightning Pose paper for technical details).
+
+First you will need to determine which videos you would like to extract frames from.
+Next, you will need to run inference on those videos with a given model in the "Train/Infer" tab;
+see the :ref:`inference documentation <tab_train_infer__infer>`.
+
+After you have completed inference you can return to the "Extract Frames" tab and select the
+appropriate option from the list:
+
+.. image:: https://imgur.com/WdXELrk.png
+
+You will then be able to select which model you would like to use (which should be the same model
+you used to run inference in the "Train/Infer" tab).
+Once the model is selected you will see a list of all videos where inference has already been
+performed.
+Select one or more videos, and as before you may also enter the number of frames per video you
+would like to label, as well as use the slider to exclude frames from the beginning and/or end
+of the video.
+
+Click "Extract frames", and you will quickly see the green message informing you that your frames
+are ready for labeling.

docs/source/tabs/train_infer.rst

@@ -6,7 +6,7 @@ Train/Infer
 
 This tab is the interface for training models and running inference on new videos.
 
-.. image:: https://imgur.com/GXhvqXI.png
+.. image:: https://imgur.com/uU6LbBZ.png
 
 The left side-bar displays your current labeling progress, and contains a drop-down menu showing
 all previously trained models.
@@ -81,6 +81,8 @@ The progress bar will reset and display inference progress for each video.
 Once training is complete for all models you will see
 "Training complete; see diagnostics in the following tabs" in green.
 
+.. _tab_train_infer__infer:
+
 Predict on New Videos
 =====================
 
@@ -91,11 +93,24 @@ You will see an upload progress bar.
 .. image:: https://imgur.com/MXHq8hx.png
     :width: 400
 
+You may also choose to create videos overlaid with predictions from the model.
+
+.. image:: https://imgur.com/RBqSZTF.png
+    :width: 300
+
+The option "Save labeled video (30 second clip)" will find the 30 second portion of the video
+with the highest motion energy in the model predictions.
+This is a good option if you want to quickly get a sense of how well the model is performing.
+
+The option "Save labeled video (full video)" will plot predictions for the duration of the entire
+video.
+This is a good option if you want to search over longer or more diverse periods of the video.
+
+If you check one or both boxes, you will be able to view the resulting videos directly in the app
+in the :ref:`"Video Player" tab <tab_video_player>`.
+
 Click "Run inference" once the video uploads are complete,
 and another set of progress bars will appear.
-After inference is complete for each video a small snippet is extracted
-(during the period of highest motion energy)
-and a video of raw frames overlaid with model predictions is created for diagnostic purposes.
 
 .. image:: https://imgur.com/rK2d7ph.png
     :width: 400

docs/source/tabs/video_player.rst

@@ -0,0 +1,17 @@
+.. _tab_video_player:
+
+############
+Video Player
+############
+
+.. image:: https://imgur.com/n2A6tBO.png
+    :width: 600
+
+This tab allows you to view labeled videos produced after running inference.
+
+On the left panel, select a model from the drop-down menu.
+Below that, you will see a drop-down menu listing all available labeled videos for that model.
+Once you select a video it will appear in the player in the center.
+The video is overlaid with a timestamp that contains the time and frame number.
+You can also save the currently displayed video to your local computer by clicking on the three
+vertical dots on the bottom right corner of the player.

docs/source/using_the_app.rst

@@ -74,6 +74,7 @@ remember that ``demo_app.py`` and ``labeleing_app.py`` only utilize a subset of
    tabs/train_status
    tabs/labeled_diagnostics
    tabs/video_diagnostics
+   tabs/video_player
    tabs/fiftyone
 
 **Close the app**