Data Augmentation update

[torzdf]

Updates the data augmentation pipeline to clean up some bugs and add some optimizations.

Bugs fixed:

• Issue with preview needing to be refreshed twice to update the images
• Issue where multiple threads would read/write to the same memory pointer, leading to image corruption
• Fixes an issue where warp/warp to landmarks would behave differently depending on training image size
• Fixes an issue where masks would sometimes be slightly misaligned

Optimizations:

• Minor optimizations across the whole augmentation pipeline
• Move more processing into the caching stage (slows down caching, but speeds up subsequent iterations)
• General refactoring to make the code more maintainable

Some numbers:
Test data (1043 A images, 1229 B images). 1000 iterations

• 512px training image size, 64px model output size, 64 batch size: ~~65% faster (including caching)
• 684px training image size, 512px model output size, 8 batch size: ~15% faster (including caching)

Summary of updates:

• lib.detected_face
  • Subclass Masks for Landmark based masks
  • Add training mask propery + methods to DetectedFace
• lib.training_training
  • subclass TrainingDataGenerator for training and preview data
  • Split cache into own module
  • Reduce thread count to 1 to prevent image corruption + data re-use
  • Process on largest model input/output size rather than stored image size
  • Size and crop masks during caching stage
  • Implement ring buffer for data flow
  • Fix preview reload bug
• augmentation
  • typing
  • switch color aug order
  • better initialization
  • Fix warp + landmark warp to correctly apply at different image scales
  • Slightly improved warp caching
  • Don't store whether image is_preview. Handle all data as training images implicitly
• plugins.trainer: Typing and fixes to work with training data refactor