Section 5 Calibration and Validation

After processing all of the data, the model was fitted using jags. The observation dataset was split into 80% for calibration, 20% for validation.

5.1 Parameter Estimates

5.1.1 Fixed Effects

Figure 5.1 and Table 5.1 present the estimated mean and 95% credible region interval (CRI) of each fixed effect parameter. The intercept term is not shown in the figure because the values are much larger than the other parameters, and would thus skew the scale.

Estimated Mean and 95% CRI of Fixed Effects

Figure 5.1: Estimated Mean and 95% CRI of Fixed Effects

Table 5.1: Estimated Mean and 95% CRI of Fixed Effects
Variable Mean Lower CRI Upper CRI
intercept 16.808754655 16.64546155 16.975095561
AreaSqKM 0.375866479 0.28792929 0.464783002
impoundArea 0.344509930 0.25902956 0.435774878
agriculture -0.204776908 -0.27556633 -0.133928339
devel_hi -0.102811082 -0.16428769 -0.043793929
forest -0.453971660 -0.53417549 -0.375287667
prcp2 0.039073783 0.03710098 0.041065222
prcp30 0.043927160 0.03714748 0.050522977
prcp2.da -0.041943771 -0.04389517 -0.039938916
prcp30.da -0.081032955 -0.08811429 -0.074322510
airTemp.da 0.060420298 0.03588743 0.085377156
airTemp.impoundArea -0.081281612 -0.10721626 -0.056587015
airTemp.agriculture -0.022766098 -0.04343778 -0.002371026
airTemp.forest -0.012313360 -0.03438179 0.009809480
airTemp.devel_hi -0.007922565 -0.02477402 0.009115676
airTemp.prcp2 0.022548230 0.02075789 0.024385230
airTemp.prcp30 -0.050218118 -0.05384880 -0.046557625
airTemp.prcp2.da -0.013240318 -0.01515015 -0.011348685
airTemp.prcp30.da -0.006635430 -0.01057342 -0.002871656

5.1.2 HUC8 Random Effects

Figure 5.2 shows the estimated mean and 95% credible region interval (CRI) for each random effect and HUC8. Table 5.2 lists the estimated mean and 95% CRI of each parameter averaged over all HUC8s (mean value with standard deviation in parentheses).

Estimated Mean and 95% CRI of HUC Random Effects for Each HUC8

Figure 5.2: Estimated Mean and 95% CRI of HUC Random Effects for Each HUC8

Table 5.2: Mean and 95% CRI of HUC8 Random Effects Averaged Over All HUC8s (Mean Value and Std. Dev. in Parentheses)
Variable Count Mean Lower CRI Upper CRI
intercept.huc 140 0.000 (0.475) -0.785 (0.531) 0.784 (0.555)
airTemp 140 1.977 (0.203) 1.713 (0.243) 2.240 (0.224)
temp7p 140 1.410 (0.304) 1.040 (0.338) 1.778 (0.352)

5.1.3 Catchment Random Effects

Figure 5.3 shows the distribution of the estimated mean for each random effect term over all catchments. CRIs are not shown due to the large number of individual catchments (8541). Table 5.3 lists the estimated mean and 95% CRI of each parameter averaged over all catchments (mean value with standard deviation in parentheses).

Distribution of estimated mean for each random effect over all catchments

Figure 5.3: Distribution of estimated mean for each random effect over all catchments

Table 5.3: Estimated mean and 95% CRI for each random effect averaged over all catchments (mean value with std. dev. in parentheses)
Variable Count Mean Lower CRI Upper CRI
intercept.site 2,847 -0.001 (1.473) -0.813 (1.484) 0.812 (1.508)
airTemp 2,847 0.000 (0.363) -0.300 (0.378) 0.301 (0.382)
temp7p 2,847 0.000 (0.386) -0.524 (0.459) 0.524 (0.394)

5.1.4 Year Random Effects

Figure 5.4 and Table 5.4 present the mean and 95% CRI of the intercept term for each year. Recall that there are no random effects for years other than the intercept.

Estimated Mean and 95% CRI of Intercept Random Effect for Each Year

Figure 5.4: Estimated Mean and 95% CRI of Intercept Random Effect for Each Year

Table 5.4: Estimated Mean and 95% CRI of Intercept Random Effect for Each Year
Year Mean Lower CRI Upper CRI
1991 0.001 -0.363 0.378
1992 0.085 -0.227 0.422
1993 0.166 -0.152 0.506
1994 0.064 -0.241 0.379
1995 -0.145 -0.400 0.089
1996 -0.288 -0.529 -0.061
1997 -0.062 -0.263 0.143
1998 0.149 -0.045 0.348
1999 0.023 -0.158 0.214
2000 -0.307 -0.429 -0.185
2001 0.115 0.000 0.224
2002 -0.025 -0.139 0.082
2003 -0.073 -0.188 0.043
2004 0.114 -0.002 0.226
2005 0.117 0.001 0.232
2006 -0.139 -0.245 -0.038
2007 -0.248 -0.348 -0.145
2008 0.092 -0.012 0.197
2009 0.036 -0.072 0.139
2010 0.159 0.066 0.251
2011 -0.071 -0.158 0.020
2012 0.201 0.117 0.289
2013 0.145 0.057 0.230
2014 -0.007 -0.091 0.080
2015 -0.196 -0.282 -0.111
2016 0.213 0.127 0.301
2017 -0.163 -0.248 -0.074
2018 0.243 0.145 0.338
2019 -0.214 -0.335 -0.093

5.2 Goodness-of-Fit

5.2.1 All Observations

Table 5.5 summarizes the model goodness-of-fit for all observations in the calibration and validation datasets.

Table 5.5: Summary statistics of model calibration and validation
Calibration Validation
# Daily Observations 688,964 75,475
# Time Series 8,149 866
# Catchments 2,847 558
# HUC8s 140 96
# Years 29.0 25.0
RMSE (degC) 1.156 1.464
Mean Residual (degC) 0.071 0.090
Median Residual (degC) 0.086 0.087
Mean Absolute Residual (degC) 0.870 1.087
Median Absolute Residual (degC) 0.682 0.845
Minimum Residual (degC) -22.953 -10.295
1st Percentile Residual (degC) -2.941 -3.764
99th Percentile Residual (degC) 2.932 3.798
Maximum Residual (degC) 14.182 9.258

Figure 5.5 presents scatterplots of predicted vs. observed daily mean temperature for the calibration and validation datasets. The black line is the 1:1 line of equality. The red line is a linear regression trend line.

Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets

Figure 5.5: Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets

5.2.2 Deployments

Table 5.6 summarises the mean, median, minimum and maximum RMSE for each deployment (i.e. continuous timeseries of observations at a single location) in the calibration and validation datasets.

Table 5.6: Summary statistics of model calibration and validation RMSE for each deployment
Calibration Validation
# Time Series 8149 866
Mean RMSE (degC) 1.044 1.282
Median RMSE (degC) 0.936 1.100
Minimum RMSE (degC) 0.200 0.210
Maximum RMSE (degC) 7.681 7.419

Figure 5.6 shows the distribution of deployment RMSE.

Distribution of deployment RMSE

Figure 5.6: Distribution of deployment RMSE

5.2.2.1 Calibration Deployment Examples

Figures 5.7 to 5.8 show example deployments from the calibration dataset with the highest and lowest RMSE.

Deployments with lowest RMSE in calibration dataset and n >= 30

Figure 5.7: Deployments with lowest RMSE in calibration dataset and n >= 30

Deployments with highest RMSE in calibration dataset and n >= 30

Figure 5.8: Deployments with highest RMSE in calibration dataset and n >= 30

5.2.2.2 Validation Deployment Examples

Figures 5.9 to 5.10 show example deployments from the validation dataset with the highest and lowest RMSE.

Deployments with lowest RMSE in validation dataset and n >= 30

Figure 5.9: Deployments with lowest RMSE in validation dataset and n >= 30

Deployments with highest RMSE in validation dataset and n >= 30

Figure 5.10: Deployments with highest RMSE in validation dataset and n >= 30

5.2.3 Catchments

Table 5.7 summarises the mean, median, minimum and maximum RMSE of all catchments in the calibration and validation datasets.

Table 5.7: Summary of catchment RMSE values for calibration and validation datasets
Calibration Validation
# Time Series 2847 558
Mean RMSE (degC) 0.977 1.320
Median RMSE (degC) 0.883 1.144
Minimum RMSE (degC) 0.222 0.352
Maximum RMSE (degC) 4.691 7.419

Figure 5.11 shows the distribution of catchment RMSE.

Distribution of catchment RMSE

Figure 5.11: Distribution of catchment RMSE