-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TCRM V3 seems to have a low cp bias #80
Comments
UPDATE: |
Hi @KayShelton this is a great bit of analysis, and I am personally doing similar for the full Aus basin presently, including comparisons to potential intensity. We'll flag any modifications that may arise. If there's anything you want me to follow up on, please let me know |
When we changed to |
So I did further investigating, and the choice of KDE doesn't appear to make a great deal of difference except to the initiation day. Comparison plot for the genesis pressure: I did also investigate the presence of the 939hPa genesis pressure for a TC. Turns out there is an issue in the IBTrACS code (https://groups.google.com/forum/#!topic/ibtracs-qa/on8IlePsqaA), which means data from Nadi is missing in the WMO fields. IBTrACS folks will resolve it and update before the end of Apr. So all in all, I don't think there is an issue with the new KDE methods, but it is worth noting that they will give different results to those in TCRMV2, and they are a lot less forgiving to outlier data. I have re-run the evaluation track generation for to 500 simulations for 1981-2016 (using IBTrACS v3.10, after manually editing the WMO TC records to include obs from RSMC Nadi). The resulting minimum pressure maps compared to the previous with V3 (previous V3 left, new V3 right): The new maps show a little more definition in the pattern, but are still a long way from V2. Also, looking at the new Q-Q and PDF plots of min pressure, the distribution is more spread out than V2, and much more so than the obs. The PDF tail at lower min P values is significantly fatter (V2 left, new V3 right): |
Thanks for the detective work @KayShelton With the kernel comparison, can you reproduce for a different cell number? Curious to see if there is any spatial variability to that result. |
I have tried a possible alternative for the minCP threshold: 4. * pstat.sig[cellNum])):
After a little trial and error, I though that limiting the minCP to 1SD below the lowest observed was probably reasonable, as that is still going to be pretty low. The resulting minCP plot (note I have only done 5 simulations of 59 years for quicker processing): I think this looks a lot better. I have also added in a plot to do samples (as done for e.g. track density), and I think they look pretty decent too: One issue, however is the relatively low track densities in comparison to observations: I think this may also be feeding into the minPressureDist, where we have a quite high proportion of the distribution with minCP higher than observed (around 1000hPa). This may be due to a relatively large number of short-lived weak systems (hence the low track densities), or it may be related to the possible initEnvPressure bug I have noted in #87. It would be interesting to investigate the distribution of track duration in the synthetic dataset compared to obs. |
So, interestingly, resolving the bug in #87 does appear to have also fixed the slight bias in the minCP histogram toward higher than observed pressures. Below is the results for 500 simulations of 59 years: Note there is now a bit of a lump at low pressures, likely due to me using 1 standard deviations below the lowest pressure observed as by lower bound on the pressure. |
I have been following up on the low track densities which are evident in the evaluation simulations for the South Pacific, but potentially more problematic in regions with very few TCs, such as the Arabian Sea. To allow me to investigate this, I have copied and modified some the the scripts in the Evaluate module to look into the TC longevity (track length), as well as a few other things while trouble shooting. I have identified that TCRM does seem to have a very strong bias towards very short-lived TCs (12 hours), as shown in the TC longevity comparison below (using the 500 South Pacific evaluation simulations from above). In attempting to resolve this issue, I have also determined that TCRM may be getting the minimum central pressure distribution (shown in the post above) close to right but for the wrong reason: there are many TCs that only live for 12 hours hence have high minimum central pressures. My resolutions to this issue are something of a compromise, but I wanted to check they are physically/statistically sound:
@wcarthur do these modifications sound sensible and valid? I will re-run the South Pacific simulations to present the results of these changes. I have been testing in the NIO for another project, and this area has very limited intensity information, so may not be representative of other basins. |
Hi @KayShelton Can you provide a diff of the code that captures teh changes you've made. It will allow me to test and integrate the changes quickly into the code base. |
@wcarthur as requested a diff of TrackGenerator.py. Note I have added a fair few print statements too, to allow me to figure out what was going on. Also added is the code for the longevityDistribution (renamed to a .txt). Note this is lifted straight from pressureDistribution meaning I have not necessarily changed variable names appropriately. |
I have been trying to get v3.0.1 on a local (UK) linux virtual machine, and most things have been successful, except the results are inconsistent with v2 results.
I have repeated an evaluation simulation (500 sims of 59 years), using the same IBTrACS historical data and same other set-up parameters (as best as I can tell) as a colleague had done previously, but my results have a significant low pressure bias.
The minPressure plot created from ExecuteEvaluate is attached for V3 and V2 of TCRM. As far as I can tell, these simulations should be near identical (there are some very subtle differences in the historic data which I cannot explain, but for the most part they are near identical). Although I would not expect identical results in the simulations, I did not expect them to be so vastly different.
Investigating the individual track generation for V3, the _stepPressureChange function in TrackGenerator.py seems to give some very large pressure fluctuations, e.g. for one track with a timestep of 6 hours, the pressure changes were: -0.1, -9.0, 17.1, -9.7, -11.3, -9.1, 4.4, 3.6, 4.1. For another TC the new self.dp, in _stepPressureChange, is given as -2.13 where the sigma is only 0.633, while another has a new self.dp of -3.13, with sigma of 0.56. These do seem slightly excessive pressure changes.
Have you seen similar results?
![minPressure_V2](https://user-images.githubusercontent.com/62250896/77929498-a3b3eb80-72a1-11ea-8f07-c867e72148db.png)
![minPressure_V3](https://user-images.githubusercontent.com/62250896/77929505-a44c8200-72a1-11ea-8572-f9abcd3efe63.png)
I first tested with the IBTrACS V4, but then re-ran to repeat an old experiment when the results were not as expected. Any help or insights would be very much appreciated.
V2 results:
V3 results:
The text was updated successfully, but these errors were encountered: