GW recharge, BRF and new project management interface

New features:

• Added BRF calculation in WHAT
• Allows xlsx water level datafiles (in addition to xls) to be loaded in WHAT
• Project info are now saved in a database using the HDF5 format instead of a text file. Water level and weather datasets can now be imported in the project. Since data are not loaded from the source file anymore, this greatly improves the performance of the system. Moreover, it makes the saving of results easier.
• Added GW recharge estimation in the GUI

GUI improvements:

• Added mouse x coordinate at the bottom of the graph in computation mode
• Removed About Tab and added an icon instead

Bug fixes and performance improvements:

• Python 3.4 compatibility bug fixes
• Important improvement of the plotting of the vertical guide for adding peak to the hydrograph.
• Removed useless/annoying warning messages
• Made the loading of weather data file faster in gapfillgui
• Fixed a bug that made the calculation of the MRC fails when there was missing data in the water level time series.
• Code was cleaned and repo was restructured
• Solved a bug when space in Windows path
• It is now possible to set whether data are removed from the dataset or not in the cross validation procedure. If not, the MLR model is not redefined each day and the memory buffer is used instead, which allow to greatly speed up the process.
• water level measurement was not read when there was only one data in the waterlvl_manual_measurements.xls file
• Solved clipping issue in EMF format for weather average
• Added info about MRC units in the equation
• Removed a lot of dependancies to pyplot
• Added nan support for the calcul of ETP

Environment Canada Update

Updated the software for compatibility to the new update of the Government of Canada website.

Weather Averages Graph Corrections and Improvements:

• Corrected the ylabel for temperature in monthly averages graph ("Monthly Mean Air Temperature (°C)" for "Monthly Air Temperature (°C)").
• The color scheme is now updated if it is changed in the color menu for the hydrograph in the weather averages graph.
• Updated the sample image for weather averages.
• Linked label language changes in HydroPrint to Weather Average graph labels.

Hydrograph Improvements

• Weather re-sampling scale, date mode for x axis labels, Date labels display frequency, and label language are now saved when saving the layout of the graph.
• Vertical padding between xlabels and axis is now done in absolute units.
• Reduced bottom margin
• Split axes for air temperature from water level. Allows to have an independent horizontal grid for temperature. This is going to make easier changing Air temperature grid scale and also to hide temperature from graph to only plot water levels.
• Changed weather scale Max value from 50 to 100 mm.
• Added the option to change the height of the graph in the UI.
• Added the option in the UI to change the vertical height ratio between the top and bottom axes.
• Number of grid interval option for the lower graph can now be changed from the UI. The option has been added to the water level scale panel. This layout parameter is also now saved in the graph layout file.
• yticks for precipitation are now set dynamically to encompass the max value for total precipitation.
• Added the option in the UI to change the display pattern of the date ticks label
• Added more digits for lat/long for the weather station in data panel.

Recharge Estimation (Not yet available in the UI)

• Finished implementing GLUE for recharge estimation
• Effort in progress to plot synth. hydrograph in HydroCalc.

Bugs

• Corrected WHAT banner logo
• Corrected the code to plot only water levels without the weather.
• Correction of gridlines when meteo is not plotted in hydrograph.
• Solved a bug where water level and precipitation scale in the UI corresponded to half the scale on the graph.
• Corrected Title positioning on the hydrograph to have a fixed point offset.
• Corrected grid order to be below plots
• Corrected a Bug with yticks labels placement in Hydrograph due to an error when setting grid to False on some axes.

Bug Fix: water level datum

Bug Fix:

• Changing the datum was not working properly due to changes made in the code v4.1.10 onward.

Interface for Color Palette Management

New Features:

• Added an interface to manage the color of the artists in the Hydrographs.

Hydrograph Plotting Improvements

New Features:

• Reworked the way the graph title was displayed and worked. Now, it contains the name of the well and of the weather station and is displayed at the top left corner of the graph. It is possible to enable or disable the legend and title in the graph from the page setup menu.
• Added the legend to the hydrograph in the top right corner.
• Added the option to activate the trend on the water level. A moving average is then done on the water level. The trend is shown as a continuous line, while each data points is shown in light blue.
• Added a color class to manage the colors in the module hydrograph. Colors are now saved and loaded in a file named colors.db. It is then possible for the users to change the colors for the graph.

Bux Fixes:

• Corrected an error in the ylabel for Precipitations in french.

Bugs Fix, UI enhancements, Groundwater Recharge Estimation

Here are some of the things that were added/fixed/changed since the last release:

New Features:

• Completed the code to estimate automatically GW recharge from meteo and water level data.
• Revamp of the “plot_monthly_normals” figure. Instead of plotting only the montlhly mean daily temperature and it's standard deviation for each months, it now compute the normals for min, max, and mean daily temperature and plot the three lines in the graph.
• Added a checkbox under the menu “Advanced options” that when checked, will automatically add the estimated ETP to “.out” files when gapfilling weather data.
• Added the possibility to put the timescale in years instead of months in the hydrographs.
• The width of the hydrograph page can now be setup by the user.
• Missing weather markers are now plotted on a day by day basis on the hydrographs, instead of have a marker per bin. Missing data are now represented with a line instead of a round marker. Thide new way of plotting show better when there is a lot of missing value for a period because it is then traced with a continuous red line.
• Added the option in the Hydroprint to select the resampling window of weather data: day, week, month, and year.
• Production of graphs for the cross-validation analysis is done automatically
• A Weather Normals graph is automatically produced and saved at the end of the gapfilling procedure.
• Added the option to display the time in date format. It's possible to switch back and forth between both display option (date or Excel numeric format), but this has not been linked to the UI yet.
• Implemented the code to export weather normal in a csv.
• Added the option to show the weather normals in a table in the Weather Normals subwindow.

Performance Improvements:

• In the hydrographs, horizontal grid is now following the ticks. There is no grid at midle-length between the vertical ticks anymore.
• Now, the xticks label are aligned automatically to the ticks. Before, the alignment was hardcoded to a fixed scaling number whose value was determined visually. This means that, if the font of the labels, or the size of the figure, or the margins sizes change, the labels will remains algined correctly.
• Removed the multiple axes frames that were added for each figure. Now, there should only be one and only one frame for the entire figure.
• Made the Hydroprint module compliant with Python-3
• Huge improvement on the quality of the rendering of the image on-screen: instead of letting the QPainter do the rescaling, the QPixmap is prescaled with the 'scaledToWidth' method and store in memory, so the scaling is only done 1 time per view. The memory is cleared if a new image is loaded in the viewer.
• Made the UI more responsive for the plotting of the hydrographs.
• separated the gui from the algorithm for filling the gaps in the data. The code is now in two completely separated modules named gapfill_weather_algorithm and gapfill_weather_gui.
• Cross-Validation procedure is now done only between the specified date instead of for the entire time-series.
• In gapfill_weather_postprocess, rasterized the scatter plot because it was way to heavy to render in pdf.
• In module hydrograph, change the way saving and loading layout is done. It is now a for loop over a list instead of using a numpy array.

UI and output improvements:

• Improved the summary table look of concatenated raw data file
• Moved the checkbox “Automatically save concatenated data” to the side-panel and renamed it "Automatically save formatted data".
• In window "New Peoject", changed the 'W'and 'N' for 'West' and 'North'
• Improved the display of the current html tables across the entire application.
• Improved the display of weather station information in the “Hydrograph” tab when loading a new weather data file.
• Now a warning is issued in the console window when the waterlvl datafile is not formatted correctly.
• Now print the time elapsed during the gap-filling procedure.
• Better formatting of the info in the “err” file. Added Information also and the original data.
• Better formating of the data in “.out” file. Removed trailing zeros in the data and also formatted the date format as int instead of float.
• Added info regarding the data used for the estimation of the data in the .err file.
• In meteo/Output, output files and figure are now saved in a subfolder named after the name of the weather station instead of directly in the Output folder.
• When a missing data is not estimated because all neighboring station are empty, RMSE is now nan instead of 0.
• Added info regarding the RMSE of the MLR model and the number of data used to the .err file.

Bug Fixes:

• The “Search for station” and "Weather Averages" windows now appear in the center of the WHAT window instead of the corner of the screen.
• More robust way to load data from the “.out” files in “MeteoObj” based on condition instead of fixed position.
• Solved a bug where the cleaning of ends of dataset in class “MeteoObj” was not done properly.
• In “Hydrograph” tab, units in table for “Weather Data File” was not displaying correctly.
• Progress Bar is now reseted correctly.
• Solved an error that occurred when a search for weather station on the CDCD returned no station.
• Soved a bug that prevented the .log file to be written when there was absolutely 0 missing data. Division by 0 issue.
• In MRC calc, now the process is correctly handled if there is an error in the Min/Max sequence.
• Solved a bug that prevented the weather data file to be loaded properly when a new projet was selected.
• Corrected a bug that made the output folder incorrect when there was a “/” or “\” character in the target station name for the downloading of weather data and the gapfilling.
• Corrected a bug where French labels in hydrograph were not displayed correctly due to a missing unicode decoration to a string. This was causing the application to crash in Windows.
• Corrected a bug in the plotting of the histogram for the PDF of the precipitation.
• Solved a bug that placed the yticks labels wrong when a new weather normals graph was generated after selecting a new weather data file (the first time it was generated was ok).
• There was a bug everytime a project was loaded when the meteo Input folder was empty. The problem was that no weather_datasets_summary.log was created when the folder was empty, but the program still needed to read the file later on in the code. An empty file is now created when no weather data files are found instead.
• There was an error in the Table 1 title of the Correllation Coefficient window. It was giving year/month/month.
• When searching for weather station in the canadian database, the minimum number of year with data calculated was understimated by 1, example: 2010-2007= 3 years when it was supposed to be 4 years.
• When the datum is changed to masl (meter above sea level), the altitude ofsett given to the y axis labels is rounded so that trailing zeros on the well altitude are not carried on to the y axis labels.
• When the datum is changed, the UI Wlmin value in the spinbox widget is correctly updated.
• There was a bug with the load/save functionality for graph layouts in Windows when there was unicode characters either in the path of the meteo data file, in the well name, or in the graph title.

v4.1.6-beta: Solving Bugs, UI developpement, code cleaning and commenting

New features

• Estimation of recharge from the Master Recession Curve is done automatically and the results are displayed in the UI.
• Weather stations are now displayed within a Table instead of a drop-down menu. This allow a global view of the data that are available for download for a given study area. The weather station list can now be modified from the UI and the download of weather data can be done in batch for more than one selected weather station.
• When searching for weather station on the CDCD, results are displayed in a table. Stations can be checked to be added to the current list of stations.
• Added a new search criteria for the search of weather stations in the CDCD. Now possible to exclude station which have less than a certain number of years of data.
• Added a browse functionality to the “display weather data info” window in the right panel that display missing data information about raw data that were downloaded or loaded manually in the UI. It is also possible to save previously concatenated file at any moment from the UI.
• Changed the way the weather station list is managed. The list is not a class variable anymore. The information is all stocked within the QTableWidget and is being grabbed from it when needed. The list of weather station is not automatically and exclusively loaded from the file name “weather_stations.lst” anymore. When WHAT UI starts, the weather station list is empty and the user needs to load an already existing one or search for stations in the CDCD. The list is also not automatically saved. It needs to be explicitely done by the user.
• Revamp of the "About" tab.

Performance improvements:

• Datafile were not opened correctly in some places within the code. This could have cause some unecessary memory load in the past.

UI

• New icon for MRC calculation
• More feedback when searching for weather station on the online Canadian Daily Climatic Database.
• New icon for removing station from the list.
• New icons for “Fill” and “Fill All” button.
• New interface for "Download Data" and "Fill Data" tabs.

Bug

• Corrected wrong uses of "open" when opening file into memory.
• Solved a bug when the button "save" in the right panel of "Download Data" tab was clicked if there was no data to save yet.
• Corrected a bug when the WHAT project file path contained “\” (from a Windows machine) and was opened in the Unix like machine.
• Corrected a bug when the charatec “/” or “\” in the station name may not have been replaced correctly in the filename by “_”. Replaced all three occurances of use of string.maketrans by “mystring”.replace()

On the way to Groundwater Recharge Estimation

New Features:

• Reorganized the code to be able to change the bin width of weather data in the hydrograph figure. This feature is now present in the code, but not yet linked in the UI.
• Made the action to create a new project an explicit task independent from the opening of an already existing project. This is to prevent the creation of files and folders by mistake all over the place.
• Added the option to show water level data as dots in the "computation" mode of the tab "Hydrograph".
• Added an option to change the datum of the water level time-series from ground surface to sea level.
• Possibility to load and plot soil stratigraphic layer in "computation mode" (Somewhat bugged for the moment).
• Recharge Calculation from Master Recession Curve (Not linked to UI yet).
• Improvement of the graph Weather Normals.

UI enhancements:

• Added a new window for the creation of new project that allows the used to input various information about the project that are saved in a ".what" file. ".what" files are now required to open a project in WHAT.
• Increased the heigh of the Menu Bar.
• More flexibility in the "waterlvl" input files format. Headers can now have any numbers of row.

Performance improvements:

• Improve the loading of waterlvl data file into memory. It is now around 8 times faster
• Optimized the way weather data bin were calculated. 2 times more efficient and now allow for a bin with a variable width. Will soon add the option in the UI. This is a step towards being able to have tick marks on an other time scale than month.
• Huge improvement for the computation of the MRC with a modified Gauss-Newton method.

Graph Design:

• Added an option to disabled and change the type of gridlines used for the production of the hydrographs.
• Added an option to plot the waterlvl without the weather.

Bug Fix:

• Solved a bug that prevented the saving of the "weather normals" figure when an extension was explicitly added at the end of the file name before saving. This problem was more apparent for Windows user since Windows automatically add a file extension to the file name in its save file dialog window.
• Solved a bug in “hydroprint” that prevented the water level to be plotted when “trend_line” option was set to 1 in the “graph_layout.lst” file.
• Solved a bug when the “calculate MRC” button was clicked and not min-man were selected previously.

Performance and UI improvements for Windows 7

(1) Used fixed font size and icon instead of system default.
(2) Significant performance improvement in the interactive plotting in "computation" mode.
(3) Various UI improvements, code cleaning and commenting.

MRC calculation (experimental) + Hydrograph dynamic display.

(1) UI Improvment in tab "hydrograph": Now displayed in bitmap and added ZoomIn and ZoomOut capability with CTRL-mouse_wheel, and Pan with mouse drag.
(2) Display of the hydrograph is now dynamic and there is no need to refresh manually the figure each time a modification is made.
(3) Bug Fix: Title placement in hydrograph.
(4) UI Improvement.
(5) Added the experimental feature to calculate the MRC from the water level time series.
(6) Improvement of the UI of the window weather yearly and monthly averages.
(7) Hide ProgressBar when it is not needed.
(8) Possible to adjust the Ptot scale in the hydrograph.
(9) In tab "Hydrograph", it is now possible to toggle a "layout" mode in order to produce a publication quality figure of an hydrograph and a "computation" mode to explore and make calculation with the data.

Known Issue: In mode "computation", which is a new experimental feature added with this version, graphic display is slow in Windows 7. Will be fixed in the next release.