Section 5 Calibration and Validation

After processing all of the data, the model was fitted using jags. The observation dataset was split into 80% for calibration, 20% for validation.

5.1 Parameter Estimates

5.1.1 Fixed Effects

Figure 5.1 and Table 5.1 present the estimated mean and 95% credible region interval (CRI) of each fixed effect parameter. The intercept term is not shown in the figure because the values are much larger than the other parameters, and would thus skew the scale.

Estimated Mean and 95% CRI of Fixed Effects

Figure 5.1: Estimated Mean and 95% CRI of Fixed Effects

Table 5.1: Estimated Mean and 95% CRI of Fixed Effects
Variable Mean Lower CRI Upper CRI
intercept 16.753 16.587 16.924
AreaSqKM 0.414 0.324 0.507
impoundArea 0.302 0.214 0.387
agriculture -0.218 -0.293 -0.145
devel_hi -0.093 -0.152 -0.037
forest -0.457 -0.535 -0.380
prcp2 0.041 0.039 0.043
prcp30 0.040 0.033 0.048
prcp2.da -0.045 -0.047 -0.043
prcp30.da -0.087 -0.094 -0.080
airTemp.da 0.064 0.038 0.089
airTemp.impoundArea -0.085 -0.110 -0.061
airTemp.agriculture -0.018 -0.039 0.002
airTemp.forest -0.012 -0.034 0.010
airTemp.devel_hi -0.008 -0.024 0.008
airTemp.prcp2 0.022 0.020 0.024
airTemp.prcp30 -0.055 -0.059 -0.051
airTemp.prcp2.da -0.014 -0.016 -0.012
airTemp.prcp30.da -0.014 -0.018 -0.010

5.1.2 HUC8 Random Effects

Figure 5.2 shows the estimated mean and 95% credible region interval (CRI) for each random effect and HUC8. Table 5.2 lists the estimated mean and 95% CRI of each parameter averaged over all HUC8s (mean value with standard deviation in parentheses).

Estimated Mean and 95% CRI of HUC Random Effects for Each HUC8

Figure 5.2: Estimated Mean and 95% CRI of HUC Random Effects for Each HUC8

Table 5.2: Mean and 95% CRI of HUC8 Random Effects Averaged Over All HUC8s (Mean Value and Std. Dev. in Parentheses)
Variable Count Mean Lower CRI Upper CRI
intercept.huc 139 -0.000 (0.533) -0.815 (0.602) 0.817 (0.602)
airTemp 139 1.962 (0.216) 1.691 (0.258) 2.232 (0.233)
temp7p 139 1.395 (0.293) 1.031 (0.325) 1.758 (0.338)

5.1.3 Catchment Random Effects

Figure 5.3 shows the distribution of the estimated mean for each random effect term over all catchments. CRIs are not shown due to the large number of individual catchments (8046). Table 5.3 lists the estimated mean and 95% CRI of each parameter averaged over all catchments (mean value with standard deviation in parentheses).

Distribution of estimated mean for each random effect over all catchments

Figure 5.3: Distribution of estimated mean for each random effect over all catchments

Table 5.3: Estimated mean and 95% CRI for each random effect averaged over all catchments (mean value with std. dev. in parentheses)
Variable Count Mean Lower CRI Upper CRI
intercept.site 2,682 -0.001 (1.443) -0.814 (1.458) 0.812 (1.475)
airTemp 2,682 -0.000 (0.361) -0.302 (0.375) 0.301 (0.379)
temp7p 2,682 -0.000 (0.362) -0.513 (0.426) 0.513 (0.374)

5.1.4 Year Random Effects

Figure 5.4 and Table 5.4 present the mean and 95% CRI of the intercept term for each year. Recall that there are no random effects for years other than the intercept.

Estimated Mean and 95% CRI of Intercept Random Effect for Each Year

Figure 5.4: Estimated Mean and 95% CRI of Intercept Random Effect for Each Year

Table 5.4: Estimated Mean and 95% CRI of Intercept Random Effect for Each Year
Year Mean Lower CRI Upper CRI
1991 -0.149 -0.507 0.182
1992 0.096 -0.224 0.449
1993 0.176 -0.148 0.527
1994 0.039 -0.253 0.329
1995 0.015 -0.283 0.337
1996 -0.121 -0.427 0.165
1997 0.084 -0.117 0.295
1998 -0.032 -0.247 0.171
1999 0.248 0.038 0.472
2000 -0.318 -0.443 -0.200
2001 0.119 0.004 0.227
2002 -0.003 -0.116 0.111
2003 -0.086 -0.202 0.030
2004 0.095 -0.017 0.209
2005 0.082 -0.034 0.198
2006 -0.123 -0.229 -0.020
2007 -0.209 -0.310 -0.111
2008 0.080 -0.024 0.185
2009 -0.009 -0.108 0.089
2010 0.105 0.008 0.193
2011 -0.109 -0.201 -0.021
2012 0.154 0.066 0.239
2013 0.115 0.024 0.201
2014 -0.080 -0.169 0.003
2015 -0.260 -0.351 -0.178
2016 0.186 0.094 0.271
2017 -0.256 -0.346 -0.170
2018 0.158 0.051 0.261

5.2 Goodness-of-Fit

5.2.1 All Observations

Table 5.5 summarizes the model goodness-of-fit for all observations in the calibration and validation datasets.

Table 5.5: Summary statistics of model calibration and validation
Calibration Validation
# Daily Observations 620,238 71,136
# Time Series 7,464 785
# Catchments 2,682 522
# HUC8s 139 91
# Years 28.0 24.0
RMSE (degC) 1.141 1.423
Mean Residual (degC) 0.068 0.086
Median Residual (degC) 0.082 0.076
Mean Absolute Residual (degC) 0.859 1.068
Median Absolute Residual (degC) 0.676 0.833
Minimum Residual (degC) -22.958 -9.879
1st Percentile Residual (degC) -2.885 -3.534
99th Percentile Residual (degC) 2.896 3.928
Maximum Residual (degC) 12.384 11.961

Figure 5.5 presents scatterplots of predicted vs. observed daily mean temperature for the calibration and validation datasets. The black line is the 1:1 line of equality. The red line is a linear regression trend line.

Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets

Figure 5.5: Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets

5.2.2 Deployments

Table 5.6 summarises the mean, median, minimum and maximum RMSE for each deployment (i.e. continuous timeseries of observations at a single location) in the calibration and validation datasets.

Table 5.6: Summary statistics of model calibration and validation RMSE for each deployment
Calibration Validation
# Time Series 7464 785
Mean RMSE (degC) 1.027 1.277
Median RMSE (degC) 0.919 1.100
Minimum RMSE (degC) 0.206 0.244
Maximum RMSE (degC) 7.410 8.003

Figure 5.6 shows the distribution of deployment RMSE.

Distribution of deployment RMSE

Figure 5.6: Distribution of deployment RMSE

5.2.2.1 Calibration Deployment Examples

Figures 5.7 to 5.8 show example deployments from the calibration dataset with the highest and lowest RMSE.

Deployments with lowest RMSE in calibration dataset and n >= 30

Figure 5.7: Deployments with lowest RMSE in calibration dataset and n >= 30

Deployments with highest RMSE in calibration dataset and n >= 30

Figure 5.8: Deployments with highest RMSE in calibration dataset and n >= 30

5.2.2.2 Validation Deployment Examples

Figures 5.9 to 5.10 show example deployments from the validation dataset with the highest and lowest RMSE.

Deployments with lowest RMSE in validation dataset and n >= 30

Figure 5.9: Deployments with lowest RMSE in validation dataset and n >= 30

Deployments with highest RMSE in validation dataset and n >= 30

Figure 5.10: Deployments with highest RMSE in validation dataset and n >= 30

5.2.3 Catchments

Table 5.7 summarises the mean, median, minimum and maximum RMSE of all catchments in the calibration and validation datasets.

Table 5.7: Summary of catchment RMSE values for calibration and validation datasets
Calibration Validation
# Time Series 2682 522
Mean RMSE (degC) 0.967 1.348
Median RMSE (degC) 0.876 1.162
Minimum RMSE (degC) 0.226 0.318
Maximum RMSE (degC) 3.980 5.708

Figure 5.11 shows the distribution of catchment RMSE.

Distribution of catchment RMSE

Figure 5.11: Distribution of catchment RMSE