Updated code May 2021
- Executive Summary
- Folder Structure
- Introduction
- Data Collection
- Phase 1 - web scraped images
- Phase 2 - images from video
- Deep Learning Models
- YOLO v5
- Phase 1 - web scraped images
- Phase 2 - images from video
- Model Metrics
- Conclusions
- References
Link to my article on Medium.com "Applying Deep Learning to Environmental Issues" published August 2020
In the Pacific Northwest, salmon are vital to commerce and to the marine ecosystem, and fish population estimates are key factors in many policy decisions. Current methods require trained biologists to observe the fish passing a viewing window and manually record the fish count on a daily basis.
This project explored the possibility of using machine learning methods of object detection and classification to support these efforts, potentially enabling the collection of data in more locations and over longer time periods.
Custom trained models (e.g. YOLO v5) using images from fish ladders showed that accurate fish detection is promising but non-trivial, counting fish in a still image does not solve the problem of counting fish in video, and that classifying fish by species requires excellent viewing conditions.
|- README.md (this file)
|- assets (images used in README.md)
|- code (Jupyter notebooks)
| |- 01_Finding_Images.ipynb
| |- 02_Model_Training_YOLOv5 (Google Colab notebook)
| |- 03_Model_Inference_YOLOv5 (Google Colab notebook)
| - presentation
|- Deep_Learning_Counting_Fish.pdf (presentation slides)
Salmon life cycles follow a predictable pattern: hatch in freshwater, migrate to the ocean for the majority of their lives, and then migrate back to their original freshwater hatch sites before they spawn and then die. The time spent in freshwater and ocean salt water depends on the species.
Salmon populations in the waters of Puget Sound are estimated each year when a mature portion of the salmon migrate back from the ocean to freshwater to spawn. In many areas, this pathway is partially obstructed by boat locks (Seattle), or hydroelectric dams (Bonneville) and the salmon travel through carefully built fish ladders on this upstream journey. As they pass through the ladders, viewing windows allow them to be seen by both tourists and biologists, and human viewers are still the primary way to count the fish.
| Fish ladder | Bonneville Dam public window | Bonneville counting window (non-public) |
|---|---|---|
 |
 |
 |
| Watch at https: //www youtube com/watch?v=sabk7Khq0kQ | Photo: (c) Phillip Colla OceanLight.com |
Once tallied, the estimated population for each species determines sport fishing limits such as the number of fish per day and the length of the fishing season. This data is also used to make decisions in the operation of salmon fisheries, commercial fishing, restaurants, and tourism.
| Columbia River Chinook Season | Ballard Locks 2020 Sockeye Counts |
|---|---|
 |
 |
| News source: https://wdfw.wa.gov/news/summer-chinook-salmon-fishing-open-july-much-columbia-river | Updated count: https://wdfw.wa.gov/fishing/reports/counts/lake-washington#sockeye |
The salmon counting task is easier when few are in the ladder; the task is more difficult when many are returning at once. Some locations estimate the full population by counting for a set period of time each day and comparing to historical data. In other locations, 24/7 video recording enables biologists to review footage and tally the counts later; weekend tallies can take staff multiple days to catch up on counts. At some sites, interested individuals can sign up for daily notifications on the latest counts.
Over the course of 2 weeks in June 2020, an internet search found 168 usable images of fish traveling past viewing windows. Of these, the majority were taken by tourists and often feature the silhouettes of children in front of the glass. Images of official viewing windows were very difficult to find, in part because 1) they are probably not particularly interesting to most people and 2) for security reasons, the fish cam at the Bonneville Dam (Willamette Falls) has been disabled.
With the use of image augmentation, the original collection of 168 images was expanded by including horizontal flip, random adjustments to exposure (+/- 25%), and random changes to rotation (+/- 15%). The final 504 images contained 725 annotated fish (averaging 4.3 per image), and included 2 null examples of viewing windows with no fish.
| Image variation 1 | Image variation 2 | Labeled Image | Null Training Image |
|---|---|---|---|
 |
 |
 |
 |
For image classification, images need to contain a limited number of objects (preferably just one) and a machine learning algorithm will attempt to name the object in the image. All that is needed is an image and a single label, e.g. "cat" or "dog".
Object detection refers to the case where there are multiple instances of an object or when there are a variety of other objects also in the image. In this situation, the image also needs to be labelled to show where each object is located. Most algorithms use a bounding box for this.
The original 168 fish images were manually labeled using the free tool "labelImg" (see https://pypi.org/project/labelImg/) to draw the bounding boxes. Free tools from roboflow.ai (see https://roboflow.ai/) were used to perform the image augmentation. Leveraging the roboflow tools provided several additional benefits: the bounding boxes were automatically adjusted for images that were randomly rotated, and the images and annotations could be quickly exported in multiple formats for use in a variety of models.
Due to the poor quality of the images, a single class of "fish" was used.
After building the first model, images or video were requested from one of the fish counting sites in Washington. This request was granted with a video segment of fish in one of the ladders. With these new images, but without fish identification expertise, it was possible to create 3 classes:
- 'adipose' for fish having an intact and visible adipose fin
- 'no_adipose' for fish having no adipose fin
- 'unknown' for fish only partially in the viewing window, or whose adipose fin region is obscured by another fish or artifact
"You Only Look Once". YOLO is a popular object detection machine learning model introduced in 2015 by a group of researchers at the University of Washington. Rather than pass an image classifier multiple times over an image to see if there was, say, a dog at the upper left, or maybe at the upper right, this new approach replaced the final layers of an image classifier with additional convolutional layers that allowed it to find all instances in one pass. The immediate improvement in speed was a major leap forward for computer vision and object detection. Since the original paper, the model has been improved several times with Version 5 being released in June 2020.
Given the popularity, speed, and accuracy of YOLO, the YOLO v5 model flow available through roboflow.ai was an obvious choice. Earlier YOLO versions have keras and tensorflow implementations and can be run on a variety of hardware. At this time, only a PyTorch version of YOLO v5 has been built. This version leverages the computational speed and efficiency of a GPU for excellent results, and there are a number of examples available in blog posts and in github. For this project, the Google Colaboratory template from roboflow.ai was used. This template configures the environment and builds the model, so a simple customization consists of uploading a new training set and selecting the number of epochs for training. Once trained, the confidence threshold can be adjusted before making predictions.
For this first model, it became apparent that labeling the fish by species was going to be highly problematic. First, identification is a challenge. Sport fishermen are discouraged from identifying fish by side view alone as this can be misleading; they are instead instructed to observe inside the mouth and to look at the color of the gum line. In cloudy, poorly lit conditions, other features such as silver highlights on the tail or where the black spots are located are very difficult to see. Second, training a model to recognize fish by species requires properly labeled images, and there were no fish experts working on this project. In lieu of counting by species, the project was scaled back to count them all as 'fish'. As a first pass, even this highlighted plenty of issues.
| Sockeye | Chinook | Coho | What's this? |
|---|---|---|---|
 |
 |
 |
 |
All images (and images of other salmon species) are from http://www.eregulations.com/washington/fishing/salmon-identification/ .
Scaling back to just finding a generically labeled "fish", and scaling up to cloud based GPU resources with the latest YOLO v5 model, the results were mixed and not ready to replace a fisheries biologist.
| Impressive | Humorous | How many fish are really in here? |
|---|---|---|
 |
 |
 |
| Photo: (c) 2014 Rudy Owens | ||
| Note 1 | Note 2 | Note 3 |
Notes:
- Impressive find: Even so, an unobstructed view is an easy improvement, and this is the case for the non-public viewing windows.
- Humorous finds: This phantom floor fish is reminiscent of the light-dark patterns of the Viola-Jones algorithm, possibly providing a clue to what may be happening. A change in lighting may result in model improvements. Note the careful arrangement of can lights in the Bonneville viewing window shown previously.
- Challenges: Height and width of window are not issues, but the depth of the tank is a problem.
In terms of model metrics, the graphs below show the results of 100 training epochs (blue) and progress on 1000 training epochs (orange). In the 1000 epoch case, the model stopped making significant improvements before all training epochs were completed. Note: as of July 2020, YOLO v5 does not have the ability to save the best model.
For the second model, images seemed crystal clear by comparison, so the classes were upgraded to adipose, no_adipose and unknown. For simple images, the model worked very well. It struggled when conditions were crowded and stronger shadows appeared.
| Easy Image | Tricky Image -- extra fish | Tricky Image - missing fish |
|---|---|---|
 |
 |
 |
In the example above with the 'extra' fish label, this problem is easily solved in production by adjusting the threshold to be > 0.40; this would exclude the box with the confidence of 0.42. For now, the threshold on the model is set lower to provide information on where the model needs to be improved.
| Phase | Precision | Recall | mAP 0.5 |
|---|---|---|---|
| 1 |  |
 |
 |
| 2 |  |
 |
 |
- Precision is the accuracy of the positive predictions (TP / TP+FP) or "If you say it's a fish, what percentage of the time is it really a fish?"
- Recall is the true positive rate (TP / TP+FN) or "If there's a fish in there, what percentage of the time do you find it?"
For object detection, precision and recall are similar to their definitions in other types of machine learning. However, there is an additional consideration here, best illustrated with an example. Let's say the image has a single fish, and the model finds a single fish but draws the box on the floor. Would we want to call that a success? Or, what if it draws the box around only the head of the fish but not the body? Do we want to give it partial credit? The solution to this dilemma is something like partial credit, where the amount of overlap between the box drawn and the expected box determines the mAP (mean average precision). The most common metric here is "50% overlap", or mAP@.5. So, if we count the boxes where the model's box overlaps the label box by at least 50%, this model built with Phase 1 images was providing correct answers in roughly 50% of the cases, and the model rebuilt with Phase 2 images was providing correct answers about 70% of the time.
Based on the results from YOLO v5, salmon counting by object detection is definitely possible, and there also remain several challenges to be solved. These challenges include:
- Viewing windows with excellent lighting are required
- Viewing window height and width are not critical, but the depth needs to be carefully selected to reduce the number of fish that can obscure other fish
- Correct species labels are required for training a model to separate sockeye, chinook, and coho in addition to other species
From personal experience (easily confirmed by watching online videos), salmon swimming upstream in a fish ladder pause to rest for varying amounts of time. In some cases, they will swim slowly and maintain position, and at other times they will slow to the point that they drift backward with the current. This adds an additional level of complexity that will require an advanced system to track objects (fish) from one video frame to the next.
Video counts will require
- Ability to track an individual fish regardless of forward or backward movement
- Only a single count irrespective of the amount of time a fish remains in the viewing window
- Special thanks to Team Roboflow.ai
- Special thanks to the staff at Chelan County PUD
- https://www.nps.gov/olym/learn/nature/the-salmon-life-cycle.htm
- https://youtu.be/zoHpE5scs2I
- http://www.fpc.org/currentdaily/HistFishTwo_7day-ytd_Adults.htm
- https://wdfw.wa.gov/news/washingtons-salmon-seasons-tentatively-set-2020-21
- http://pweb.crohms.org/tmt/documents/fpp/2020/final/FPP20_02_BON.pdf
- https://idfg.idaho.gov/fish/chinook/dam-counts
- http://www.cbr.washington.edu/dart/overview
- http://www.cbr.washington.edu/dart/query/adult_daily
- http://www.eregulations.com/washington/fishing/salmon-identification/
- https://www.chelanpud.org/environment/fish-and-wildlife/fish-counts
- https://pypi.org/project/labelImg/
- https://roboflow.ai/
- https://arxiv.org/abs/1506.02640
- https://towardsdatascience.com/how-to-train-a-custom-object-detection-model-with-yolo-v5-917e9ce13208
- https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework
- Hands-On Machine Learning with Scikit-Learn and TensorFlow by Aurélien Géron (O’Reilly). Second edition. Copyright 2019.
Presentation Materials (slides)