-
Notifications
You must be signed in to change notification settings - Fork 0
/
TODO.txt
121 lines (113 loc) · 8.26 KB
/
TODO.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
TODO:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~General~~~~~~~~~~~~~~~~~~~~~~~~~
ensure ENVI header b.b. is updated if using UAVSAR clip;
nudge to get alignment bw UAVSAR and training data,
look at rasterizing alg and make sure it's not adding extra space to the training area;
*erosion of training data in arc/gdal,
set correct output path for training image stack/clip;
output txt file to list orig file
name and number-class lookup;
change output of training image to be in Train directory,
why are training and veiwing images slightly offset? resampling boundaries?
(use NoData mask for training data: https://gdal.org/development/rfc/rfc15_nodatabitmask.html#rfc-15, but will need to copy mask_valid_pixels to XXX.msk); (convert to int16 precision to save memory);
save arcmap viewing color files...;
add cost functions...;
*Run training for lots of param options and figure out which ones most imp- then use those
* Try decomp with no moving (boxcar) window filter
*Range correction and inter-year differences.(ENVI histogram?)
Run in regression mode to determine optimal class divisions?
Test effect of not using incidence angle as a feature to remove potential range-of-training-area bias
Update colors to be color-blind friendly
Near-and far range masking or simply save output file w range band
Add tall shrub class!
* dynamically assign mean incidence angle, or use lookup table from metatdata.ann file
Auto/streamlined env.input(xx) ingest
**Add H/a/alpha or Sinclair matrix or AVIRIS into workflows
*Illum angle correction for shield
RE-write parallel_cat_run.sh as a slurm batch file that uses srun instead of bash to call the processing function
Add aliases to view most recent slrum-###.out file. Create standard output files a la https://gist.github.com/zkamvar/3d1fa899db5eecb1295de4cd84052a7b
*Parallelize trainingImageimport.m - use parfor to run all the gdal tasks at once - but this will require renaming structures and setting different gdal CACHEMAX
Follow-up to Zhange et al 2013 IJRS: invesitage effects of relative class size on RF training. Make same N vs OA plots, but with among-polygon training/val splits.
alt accuracy metrics - using just 3 classes
From RTC: Change my ENVI header bash script in polsar_pro_wkflow to use python script instead
If I add more files, but don't compute Freeman: modify lines 70-90 in trainingImageImport.m to change filename indexing (no Freeman will be presnet)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~PixelClassifier~~~~~~~~~~~~~~~~~~~~~~~~~
TODO: change plotting for n-band images;
test w multiple imgs in training dir (maybe set to load indiv training image sets into mem one at a time);
mirror window in movingstd instead of shrink...
k-fold cross val for number of leaves?
Memory optimization/block proc, if necessary
Make addOutputImages.m more general by removing function calls to env (and delte .png files)
Community service: Update fileexchange script for the following with gif fix: D:\Dropbox\Matlab\DownloadedCode\CaptureFigVid\CaptureFigVid\CaptureFigVid_EK.m
Delete class mask images in addOutputImages.m as cleanup
make mem efficient by possibly modifying way imClassify.m saves subsets{i} all the same
Re-write to only compute 4 sigmas, not >=8
Possible LOO cross val on n-value for range correction, and try non-cosine as well
try adding in NaNs, but removing NaNs just before steerable detector to save compute time...
save uncertainties of each class by normalizing class probs by total prob
More radiometric cal: I would plot sigma, gamma and Freeman values based on inc angle and eval if I need to improve my method.
Then, I could come up with empirical correction, or use Chao's values.
I could also implement the Ulander 1996 illum. area correction
Run with only raw Freeman image ratios or inter-class variance
view(treeBag.Trees{1}, 'Mode', 'graph')
* Rewrite for large datasets or chunk up YF lines (auto change chunk size if isunix) - pre-compute filters, then load in chunks - Re-write using blockproc with padding to run in chunks and save memory. Or optimize for images with large NoData areas. - Change to not load all files in dir. into memory!
* Add histogram matching or use diff decomp (Sinclair) so as to accurately compare between seasons/years if bad calibration... yFlats vals were low
* Radiometric terrain correction / MapReady.... redo entire workflow...
* Improve hot fix that undid class size normalization using random number generator
** Do range correction before decomp...perhaps using JPL/Marc workflow
** Change error assessment to randomly select by polygon, not by pixel, so as to produce more realistic metrics
TODO: modify original code for parallel to not load entire mat-obj of F into mem (imClassify)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~LINUX~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
add ability to sbatch each test file in parallel! Need command line input for which files to run on
X make range correction additions linux-compatible
set data dir input files to parse out beginning paths and _grd so I can use same for both scripts
X Fix stacking bug so I can run on second Shield Image! X
parallelize training imageImport to use parfor on two cores
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ASC~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
update gdal-python to 2.1.2+ to fix gdal_edit.py error: http://osgeo-org.1560.x6.nabble.com/gdal-dev-OpenEx-not-found-gdal-edit-py-td5334123.html
(Updae Matlab to R 2019a or greater to import confusionplot() )
Ask if they have or can install Matlab Distributed Computing Server
~~~~~~~~~~~~~~~~~~~~~~~~NOTE~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
nodata value is not dynamically defined in the gdal calls
pixelClassifierTrain doesn't have if statements for non Freeman-inc input class options
~~~~~~~~~~~~~~~~~~~~~~~~DONE~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
add bounding box input to create smaller training file tif, reproject,
run with only raw Freeman images
add moving std feature
test training using 3 bands!;
modify to work on n-band images,
how to do deal w empty files for empty training classes;
automate bounding box selection
*What percentage of training px are being used? A: <1
*making confusion matrix; Multithread gdalwarp, dep on linux or Brown- but careful if running loops in parallel
re-plot histograms to better align by band
XAdd Lee filter (wiener2.m)
X Remove some of the filters from sigma: i.e. Hess
After AGU: add NoData class, use translation features just for x- to combat overlay effect?
*** Add in W2 class (removed by mistake...)!!!
* Rotate offset images and only use directions parallel to look angle (discard orthogonals)
circumvent output method of writing to sep files, then combining- just combine on the first try (in pixelclassifier.m); (save output training diagram/image to folder)
*Need better water and FD classes
* Don't classify NoData areas! Don't train with NoData areas! Don't do filters on -10000 vals that should be NoData (rm conversion from NaN)
NOTE: env.output.train_dir and env.class_dir_asc point to same thing-are redundant
* Fix NoData output issues...
Fix GeoTiffWrite alternatives for large bigtiffs
add ifunix branch to use gdaledit.py instead of gdalwarp
*Better training data! Then re-run trainingImageImport.m
noData mask for training;
turn off parallel except if on a cluster?
speckle filtering?
*Try with C3 or T3 matrix or other decomp to see if yellow overlay is improved or better class distinction.
Multithread gdalwarp, dep on linux or Brown- but careful if running loops in parallel
prep trainingImageImport.m to be like viewingImageImport.m
** Put slope into classifier?
* Fix training image bug on ASC (search for %%HERE in combine_imgs; mask is wrong size or output is NaN)
** use gdalwarp -wm 8000 to set memory! CACHEMAX may not be enough...
~~~~~~~~~~Modify for re-distribution of PixelClassifier Code~~~~~~~~~~~~~~
rfTrain.m
pixelClassifierTrain.m - validition and concat of n=3 bandns to make F
** if classifying in parallel (2 workers), make parpool(4) and call for 8 cpus, or change chunking size and num cpus during classification to avoid memory crash!
Promote biggeotiffwrite as a useful function!
~~~~~~~~~~~~~~~~~~~~~~~~Requirements~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gdal (and/or the gdal python scripts though)
topotoolbox installed and path indicated in env