The files in this repository are designed for use with Articulate Assistant Advanced, software for speech articulatory analysis and recording by Articulate Instruments.
You will also need:
- Windows 10 or later (Windows 7 or 8 work but see troubleshooting; use at your own risk).
- At least 3.6 GB free space on your hard drive to install downloaded software to.
- Analysing recorded data and real-time tracking should work on any computer, but if you want live-tracking at a high framerate and much faster data analysis then you will need a CUDA compatible GPU in your computer. CUDA is only compatible with Windows 10 and later.
NB: You do NOT need to manually download or install DeepLabCut. Continue reading this page for installation instructions.
Some shorthand terms will be used throughout this tutorial:
- "Model" = A neural net's trained memory for a specific task, ie. a standalone bundle of files and folders representing training, which can then be used to automatically plot points on new data.
- "AAA" = Articulate Assistant Advanced.
- "DLC" = DeepLabCut, the software tool for training and using models.
DeepLabCut is popular free open-source software which you can use to train machine-learning models that automatically determine the coordinates of real-world objects from images and video. Articulate Assistant Advanced interfaces with it directly so you can automatically annotate ultrasound data and video-camera data with splines just by using the normal AAA menus.
DeepLabCut can run on any computer running Windows 10 and later (see troubleshooting for running on Windows 7 and 8). If your computer also contains a CUDA GPU then DeepLabCut can use this to process data a lot faster.
In AAA you can either use DeepLabCut to spline existing recordings or use it to track anatomy live in real-time. For live tracking, a computer with an NVIDIA GPU supporting CUDA can process data fast enough to do real-time tracking at 50-60 Hz (which is the fastest that most computer monitors can display), and if you don't have a CUDA GPU then a common office PC in 2022 can do 15-25 Hz.
The lower framerate on a common office PC is fast enough to be adequate for live-tracking most tongue movements, but some particularly fast tongue movements such as trills may be too fast to live-track without a higher-end PC and/or a CUDA GPU. However, even on a budget PC recording data for later analysis will always be done at the maximum framerate of the ultrasound hardware, which depending on the system will likely be between 80 - 120 Hz.
DeepLabCut was created by Mathis, A., Mamidanna, P., Cury, K.M. et al. (2018) (10.1038/s41593-018-0209-y) with additional software by Nath, T., Mathis, A. et al. (2019) (10.1038/s41596-019-0176-0) and Mathis, A., Biasi T. et al. (2021)
DeepLabCut is a powerful but often complicated software tool. Many people struggle a lot with their first time using it. To make life easier, AAA handles most of the complicated stuff automatically so you don't have to. If you only want to fit splines to your data (either individually or as a batch process) and/or use live-tracking for oral anatomy then just follow the rest of the instructions in this document. If you want to use DeepLabCut in a more complicated way not covered by AAA's simplified interface, you can read a tutorial on using DeepLabCut's more advanced features with AAA here.
If this is your first time using DeepLabCut, you will first need to manually install a free open-source software tool called Anaconda. It is a popular data science tool which AAA will automatically communicate with to install, update and run DeepLabCut. If you already have a full version of Anaconda installed you can use that instead.
There are two versions of the Anaconda software to choose between. Miniconda is a compact, simplified version of Anaconda that takes less file space on your computer and has everything needed to run DeepLabCut in AAA. However, it lacks the broad range of data science tools and pre-installed packages that are included in the full version of Anaconda, which require a larger download and more hard-drive space. If you don't plan to use Anaconda for any other purpose than DeepLabCut in AAA then Miniconda is sufficient.
In AAA, you can open the DeepLabCut settings menu either from the Edit Splines dialog or from the Live Tracker tabs of Ultrasonic Setup or Video Setup. (These can be accessed by right-clicking on an ultrasound or video display). You will be presented with a few options: one of them will be a place where you should type in the full path to your Anaconda directory (with no trailing backslash):
If your computer contains an NVIDIA CUDA compatible GPU then your computer can run DeepLabCut up-to 4 times faster. However, to benefit from this additional performance you must complete all of the following additional installation steps:
- Install Visual Studio 2019 (it must be the 2019 version; more recent versions of Visual Studio won't help). Ensure that Visual Studio 2019 has the
Desktop development with C++Workload enabled with theMSVC v142 ...andC++ CMake Tools for Windowscomponents. If you already have a version of Visual Studio 2019 installed, you can modify it to include the Workload by opening your Windows Start menu and searching forVisual Studio Installerand running it, then modifying your existing Visual Studio 2019.
You can reduce the hard-drive space that Visual Studio 2019 takes up by deselecting certain Individual Components of the
Desktop development with C++Workload that are not required: you must keepMSVC v142 ...andC++ CMake Tools for Windowsenabled, but you can safely de-select the other components.
- Install the CUDA 11.2 drivers (it must be this version; more recent versions of CUDA won't work; follow the link and download version "10" for both Windows 10 and Windows 11).
- After successfully installing CUDA 11.2, copy these dll files into the
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\binfolder.
If you get an error: If your computer already has up-to-date Nvidia drivers installed then when you attempt to install the CUDA 11.2 drivers it may give an error saying that it refuses to install version 11.2 when there is more recent Nvidia software already on the computer; to resolve this you should uninstall the
Nvidia Frameview SDK, which you can do by opening your Windows Start menu and openingAdd or remove programs, then using the search box to search forFrameview, clicking on it when it appears, and uninstalling it. After doing this, restart your computer and then try installing CUDA 11.2 again. If this does not work, you may need to also remove any other versions of CUDA on your computer using a similar method.
If you failed to complete all the above steps correctly then DeepLabCut will still successfully analyse data but will do so by falling-back to using your CPU instead, which will likely be many times slower. You can verify if CUDA is working successfully to speed up your data processing: to do this, attempt to use DeepLabCut in AAA (for instructions on how to do so, please continue reading this document). When you begin using DeepLabCut to apply a spline or do live-tracking, a Windows Command Prompt window will appear in your taskbar at the bottom of your screen. Please click on it to view it and then read the text within it. If you installed CUDA correctly as above, then you should see a line appear part-way through the text which says:
Created device with ... memory --> device: 0, name: [Your GPU's name]
You can find the name of your GPU by opening your Windows Start menu and opening Device Manager, then within that open the subheading Display Adapters, which will list all the GPUs in your computer. If you have at least one NVIDIA device listed there then DeepLabCut will be able to use it if you performed the above steps correctly.
You must make at least one Model available to AAA or else it cannot use DeepLabCut to analyse data. Each Model is trained to perform a specific task, for example annotating tongue and mouth anatomy from mid-sagittal ultrasound images, or annotating lips from camera images captured facing the front of the mouth.
Models trained by Articulate Instruments for use in speech production analysis and visual feedback are available for free download in this repository at the top of this page. You can download individual models from Releases or clone this repository to your AAA\DLCModels folder. Over time the models available here may be updated to improve accuracy and/or performance, and new models might be added to perform different analyses.
Each model should be a folder containing:
Snapshotfiles- A
pose_cfg.yamlfile - A
AAAmodelfile
Please put each such folder inside the \DLCModels folder in your AAA directory. An example of correct folder structure is below.
If no \DLCModels folder exists you may not be running a recent enough version of AAA to support integrated DeepLabCut. Consider updating your AAA version. If you have a sufficiently recent version of AAA and the folder is still missing, you can safely create it yourself and put the model folders in it, as above.
If you have used DeepLabCut before, you can also use models which you have trained or used previously with AAA, but to use them you must first convert them to DeepLabCut-Live format. Official documentation on this can be found here. For example: if your DeepLabCut model was in C:\MyModel then inside your appropriate DeepLabCut conda environment and within iPython and after importing DeepLabCut, you could use the command deeplabcut.export_model("C:/MyModel/config.yaml", iteration=None, shuffle=1, trainingsetindex=0, snapshotindex=None, TFGPUinference=True, overwrite=False, make_tar=True) substituting the other arguments with your own settings. It would then export the model to the folder C:\MyModel\exported-models. It may require the folder \exported-models to NOT exist or the process might fail. Note that forward slashes instead of backslashes in the file-path are required; if you use backslashes it will give an error that says SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position....
At this point, you should have now installed your preferred version of Anaconda, typed its path into AAA, and installed at least one model. The final step is to tell AAA to automatically install the DeepLabCut software, and this can be done in either of two ways:
- Restart AAA and follow the instructions to install DeepLabCut that should appear at startup.
- Attempt to use DeepLabCut in AAA, either by enabling real-time live DLC tracking or by attempting to batch fit splines to an ultrasound or video recording using DLC: AAA will automatically install DLC the first time you do this if it isn't installed yet. Instructions on how to use DLC in AAA can be found below.
Once you have successfully installed all requirements using the instructions above, you are ready to use DeepLabCut!
In AAA you can only fit 2D splines using DeepLabCut due to fan splines being unsuitable for tracking lateral movement of objects. If you really want to use Fan Splines then you can easily convert 2D splines to Fan Splines (or vice versa) by opening the Edit Splines dialog and selecting the Splines tab, then selecting the spline you wish to convert and clicking the Convert button. However, doing this will destroy information. 2D splines can double-back on themselves but Fan splines cannot: converting 2D to Fan will preserve only the closest spline surface to the probe origin. Furthermore, there cannot be more or less knots than fan lines, so additional knots may need to be created at arbitrary radii, and knots beyond the range of the fan will be removed.
In AAA, there are two ways to use DeepLabCut:
- Open the
Edit Splinesdialog by right-clicking an ultrasound or video display. - Click on the
Fit Splinetab and select theDeepLabCutoption. - Click the
DeepLabCut Settingsbutton above and choose a Model from the list at the bottom. If no Models are available it may be because of an unsuccessful installation of Models. - Select the
Batchtab andChoose Recordings. - Select the
Process Batchtab and tick the checkboxes for the splines you wish for the Model to add to your recording. They will be created as splines with the same names as shown in that list. - Batch splining many recordings could take a long time if you don't have a CUDA GPU: up to twenty times longer than the duration of the recorded data being splined. When you are ready to spline your selected data, click
Process.
- Open the
Ultrasonic Setup->Display OptionsorVideo Setup...by right-clicking on a live ultrasound or video display. - Click on the appropriate
Live Trackertab and open theDeepLabCut-Livesettings. - Choose a Model from the list at the bottom. If no Models are available it may be because of an unsuccessful installation of Models.
- When you are ready to start live-tracking, select
Live UltrasonixorLive Videoon the left. AAA may pause for up to a minute while it loads a Model. You can change the Model at any time while live-tracking, and doing so will pause tracking while it changes Model. - AAA does not currently support recording data while simultaneously live-tracking, so live-tracking will be automatically disabled if you start recording.
I tried to install Anaconda3/Miniconda3 in Windows 7 and get "Error installing: Failed to create menus"
Newer versions of Anaconda3/Miniconda3 do not install on Windows 7. Go to the archive page (https://repo.anaconda.com/miniconda/) and install 'Miniconda3-4.7.12.1-Windows-x86_64.exe' instead of the latest version.
This repository provides multiple models for different purposes. Inside each model's folder is a separate README file which explains the model's purpose and behaviour. There are also multiple models for each purpose, such as multiple models for human oral mid-sagittal ultrasound: these differ in a tradeoff of accuracy and speed.
Unidirectional models are better at estimating the positions of anatomy when the image is poor, but they require you to tell AAA which direction the tongue tip is facing in the image when you use the model. Bidirectional models don't need to be told what direction the tongue tip is facing, which can help when batch processing many recordings containing a mixture of tongue-tip directions, but they sometimes produce unexpected behaviours when the image is poor.
As a general rule, ResNet50 models provide the best accuracy but at a cost of significantly slower speed analysing data. Conversely, MobileNetv2 0.35 models can analyse data at the fastest speeds but are the least accurate. MobileNetv2 1.00 is a midpoint between the other two in both measures. You can read the official DeepLabCut documentation on the different neural networks here. Exact accuracy and speed measures can be read from inside the AAAmodel file in each model's folder, by opening it with any text editor. Whenever you analyse data inside AAA using DeepLabCut, either splining or live-tracking, it will automatically update the AAAmodel file with the average analysis speed of the model(s) you experienced on your computer.
Please ensure you have downloaded the models from this repository and installed them into the same installation folder of AAA as you wish to use, by carefully following the instructions here.
The first time you tell AAA to use a DeepLabCut model it needs to load the model and initialise DeepLabCut, and this can take a long time, but should not take longer than a minute even on an old computer.
If you think something is wrong and AAA is really frozen, please check your Windows Taskbar where you might see a Windows Command Prompt running.
- If there is a Command Prompt visible: It should be displaying the output of DeepLabCut. Any messages in it are created by DeepLabCut and not by AAA. There may be important information in there which explains why it appears to not be proceeding.
- If there's not any Command Prompt visible then it might have gone really wrong: This may be a AAA bug rather than a DeepLabCut crash. Normally when DeepLabCut closes due to finishing successfully the Command Prompt will automatically close, and if there's a problem with DeepLabCut it should remain open so you can see what went wrong. If it's not visible it might be a symptom of AAA failing to start it at all. Please restart AAA and return to the DeepLabCut options dialog. This time, please check the box labelled
Show script errorsand try to repeat what you did that led to the freeze. This will force the DeepLabCut Command Prompt window to be visible and remain open no matter what happens, and it might contain helpful information. If you can't resolve the problem easily, please contact us and we can help!
This is likely because your computer is throttling your computer's processor to prevent it overheating or drawing too much power. This is something laptops often do a lot, but it's unlikely to affect desktop PCs. If you're using a Nvidia GPU there are some settings you can change in your Nvidia Control Panel or GeForce Experience (which can be found in your Windows Start menu) to prevent your computer from deliberately slowing your GPU down when it works too hard: these settings are often found under categories or headings relating to battery usage / power management. Be careful, however, because manufacturers put these settings in for good reason: you can drain a laptop's battery really fast by running its GPU at maximum power all the time.
Each model folder contains a file called AAAmodel which can be edited using any text editor. It contains comments explaining the exposed properties of the model.
You can safely change the following properties of a model:
- Display name
- How the tracked points get interpreted into AAA splines
- The colors used to distinguish each point during live-tracking
You can also change the following properties, but it is strongly advised that you do not modify them as their values are chosen to produce the most accurate tracking:
- What pixel resolution images should be resized to before being passed to DeepLabCut
- How changes of aspect ratio should be handled in the resizing operation
This is because DeepLabCut models' training is specific to an image resolution and tracking accuracy deteriorates the more you diverge from the image resolution and aspect ratio used in training.
The advantage of cloning this repo to your AAA\DLCModels folder is you can simply pull changes whenever we update the models or add new ones and they will be immediately available within AAA.
Please ensure that you clone this repository to within AAA\DLCModels and not a folder within it. For example, it should be cloned such that a model called ResNet50 would be at the path:
AAA\DLCModels\ResNet50\= GOODAAA\DLCModels\AAA-DeepLabCut-Resources\ResNet50= BAD
Links to clone are available at the top of this page.
If you are unfamiliar with using Git, you can quickly clone this repository in this way by opening a Windows Command Prompt or Powershell window and navigating it to the AAA\DLCModels folder, then typing:
git clone articulateinstruments/AAA-DeepLabCut-Resources
Once you clone it in this way, you can open this new local repo you have created in any Git software of your choice, such as GitHub Desktop, GitKraken, TortoiseGit, VSCode, etc. From there, you can easily download updates and get new models by fetching and pulling. If you only want to work from the command-line, you can open a Windows Command Prompt or Powershell window and navigate it to the AAA\DLCModels folder, then type: git pull