Section 5 Calibration and Validation

After processing all of the data, the model was fitted using jags. The observation dataset was split into 80% for calibration, 20% for validation.

5.1 Parameter Estimates

5.1.1 Fixed Effects

Figure 5.1 and Table 5.1 present the estimated mean and 95% credible region interval (CRI) of each fixed effect parameter. The intercept term is not shown in the figure because the values are much larger than the other parameters, and would thus skew the scale.

Estimated Mean and 95% CRI of Fixed Effects

Figure 5.1: Estimated Mean and 95% CRI of Fixed Effects

Table 5.1: Estimated Mean and 95% CRI of Fixed Effects
Variable Mean Lower CRI Upper CRI
intercept 16.833421530 16.67617588 16.9867338349
AreaSqKM 0.395116342 0.30627141 0.4840870247
impoundArea 0.349202949 0.26441219 0.4384439730
agriculture -0.231818023 -0.29860379 -0.1662348226
devel_hi -0.113091616 -0.17117577 -0.0548478596
forest -0.449975232 -0.52445930 -0.3751637908
prcp2 0.037811292 0.03591343 0.0395934936
prcp30 0.044143933 0.03778849 0.0505761489
prcp2.da -0.043839749 -0.04582635 -0.0418767171
prcp30.da -0.083451495 -0.09003666 -0.0767409762
airTemp.da 0.055426531 0.03046611 0.0804564704
airTemp.impoundArea -0.075351885 -0.10018257 -0.0489819699
airTemp.agriculture -0.018888556 -0.03803921 0.0001204872
airTemp.forest -0.001042650 -0.02127786 0.0203067821
airTemp.devel_hi -0.004110505 -0.02032762 0.0124983349
airTemp.prcp2 0.024049145 0.02223997 0.0257588483
airTemp.prcp30 -0.052055111 -0.05564053 -0.0485377100
airTemp.prcp2.da -0.011881446 -0.01361107 -0.0101528157
airTemp.prcp30.da -0.006500592 -0.01012480 -0.0029489054

5.1.2 HUC8 Random Effects

Figure 5.2 shows the estimated mean and 95% credible region interval (CRI) for each random effect and HUC8. Table 5.2 lists the estimated mean and 95% CRI of each parameter averaged over all HUC8s (mean value with standard deviation in parentheses).

Estimated Mean and 95% CRI of HUC Random Effects for Each HUC8

Figure 5.2: Estimated Mean and 95% CRI of HUC Random Effects for Each HUC8

Table 5.2: Mean and 95% CRI of HUC8 Random Effects Averaged Over All HUC8s (Mean Value and Std. Dev. in Parentheses)
Variable Count Mean Lower CRI Upper CRI
intercept.huc 142 -0.001 (0.475) -0.764 (0.537) 0.764 (0.550)
airTemp 142 1.989 (0.185) 1.736 (0.224) 2.241 (0.206)
temp7p 142 1.434 (0.321) 1.065 (0.362) 1.803 (0.368)

5.1.3 Catchment Random Effects

Figure 5.3 shows the distribution of the estimated mean for each random effect term over all catchments. CRIs are not shown due to the large number of individual catchments (9231). Table 5.3 lists the estimated mean and 95% CRI of each parameter averaged over all catchments (mean value with standard deviation in parentheses).

Distribution of estimated mean for each random effect over all catchments

Figure 5.3: Distribution of estimated mean for each random effect over all catchments

Table 5.3: Estimated mean and 95% CRI for each random effect averaged over all catchments (mean value with std. dev. in parentheses)
Variable Count Mean Lower CRI Upper CRI
intercept.site 3,077 0.001 (1.477) -0.799 (1.488) 0.800 (1.509)
airTemp 3,077 -0.000 (0.370) -0.300 (0.381) 0.300 (0.389)
temp7p 3,077 0.000 (0.394) -0.528 (0.467) 0.529 (0.400)

5.1.4 Year Random Effects

Figure 5.4 and Table 5.4 present the mean and 95% CRI of the intercept term for each year. Recall that there are no random effects for years other than the intercept.

Estimated Mean and 95% CRI of Intercept Random Effect for Each Year

Figure 5.4: Estimated Mean and 95% CRI of Intercept Random Effect for Each Year

Table 5.4: Estimated Mean and 95% CRI of Intercept Random Effect for Each Year
Year Mean Lower CRI Upper CRI
1991 -0.002 -0.341 0.336
1993 0.142 -0.161 0.473
1994 0.031 -0.242 0.317
1995 -0.058 -0.290 0.161
1996 -0.263 -0.505 -0.052
1997 -0.003 -0.205 0.195
1998 0.200 0.011 0.404
1999 0.101 -0.074 0.277
2000 -0.309 -0.430 -0.190
2001 0.123 0.020 0.227
2002 -0.043 -0.150 0.064
2003 -0.002 -0.112 0.110
2004 0.109 0.001 0.219
2005 0.101 -0.008 0.210
2006 -0.136 -0.232 -0.041
2007 -0.190 -0.290 -0.096
2008 0.074 -0.020 0.173
2009 0.072 -0.026 0.169
2010 0.117 0.035 0.202
2011 -0.100 -0.184 -0.018
2012 0.199 0.119 0.280
2013 0.139 0.059 0.221
2014 -0.025 -0.104 0.051
2015 -0.223 -0.303 -0.145
2016 0.164 0.082 0.244
2017 -0.211 -0.294 -0.130
2018 0.156 0.070 0.242
2019 -0.128 -0.221 -0.040
2020 -0.042 -0.139 0.059

5.2 Goodness-of-Fit

5.2.1 All Observations

Table 5.5 summarizes the model goodness-of-fit for all observations in the calibration and validation datasets.

Table 5.5: Summary statistics of model calibration and validation
Calibration Validation
# Daily Observations 749,327 85,228
# Time Series 8,495 959
# Catchments 3,077 614
# HUC8s 142 104
# Years 29.0 25.0
RMSE (degC) 1.152 1.589
Mean Residual (degC) 0.076 0.137
Median Residual (degC) 0.087 0.150
Mean Absolute Residual (degC) 0.866 1.148
Median Absolute Residual (degC) 0.678 0.883
Minimum Residual (degC) -12.129 -15.074
1st Percentile Residual (degC) -2.918 -3.875
99th Percentile Residual (degC) 2.944 4.215
Maximum Residual (degC) 13.775 14.658

Figure 5.5 presents scatterplots of predicted vs. observed daily mean temperature for the calibration and validation datasets. The black line is the 1:1 line of equality. The red line is a linear regression trend line.

Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets

Figure 5.5: Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets

5.2.2 Deployments

Table 5.6 summarises the mean, median, minimum and maximum RMSE for each deployment (i.e. continuous timeseries of observations at a single location) in the calibration and validation datasets.

Table 5.6: Summary statistics of model calibration and validation RMSE for each deployment
Calibration Validation
# Time Series 8495 959
Mean RMSE (degC) 1.043 1.345
Median RMSE (degC) 0.938 1.133
Minimum RMSE (degC) 0.189 0.152
Maximum RMSE (degC) 8.709 12.401

Figure 5.6 shows the distribution of deployment RMSE.

Distribution of deployment RMSE

Figure 5.6: Distribution of deployment RMSE

5.2.2.1 Calibration Deployment Examples

Figures 5.7 to 5.8 show example deployments from the calibration dataset with the highest and lowest RMSE.

Deployments with lowest RMSE in calibration dataset and n >= 30

Figure 5.7: Deployments with lowest RMSE in calibration dataset and n >= 30

Deployments with highest RMSE in calibration dataset and n >= 30

Figure 5.8: Deployments with highest RMSE in calibration dataset and n >= 30

5.2.2.2 Validation Deployment Examples

Figures 5.9 to 5.10 show example deployments from the validation dataset with the highest and lowest RMSE.

Deployments with lowest RMSE in validation dataset and n >= 30

Figure 5.9: Deployments with lowest RMSE in validation dataset and n >= 30

Deployments with highest RMSE in validation dataset and n >= 30

Figure 5.10: Deployments with highest RMSE in validation dataset and n >= 30

5.2.3 Catchments

Table 5.7 summarises the mean, median, minimum and maximum RMSE of all catchments in the calibration and validation datasets.

Table 5.7: Summary of catchment RMSE values for calibration and validation datasets
Calibration Validation
# Time Series 3077 614
Mean RMSE (degC) 0.970 1.436
Median RMSE (degC) 0.869 1.149
Minimum RMSE (degC) 0.281 0.394
Maximum RMSE (degC) 4.630 12.401

Figure 5.11 shows the distribution of catchment RMSE.

Distribution of catchment RMSE

Figure 5.11: Distribution of catchment RMSE