Section 5 Calibration and Validation

After processing all of the data, the model was fitted using jags. The observation dataset was split into 80% for calibration, 20% for validation.

5.1 Parameter Estimates

5.1.1 Fixed Effects

Figure 5.1 and Table 5.1 present the estimated mean and 95% credible region interval (CRI) of each fixed effect parameter. The intercept term is not shown in the figure because the values are much larger than the other parameters, and would thus skew the scale.

Estimated Mean and 95% CRI of Fixed Effects

Figure 5.1: Estimated Mean and 95% CRI of Fixed Effects

Table 5.1: Estimated Mean and 95% CRI of Fixed Effects
Variable Mean Lower CRI Upper CRI
intercept 16.780 16.614 16.953
AreaSqKM 0.398 0.307 0.490
impoundArea 0.376 0.294 0.460
agriculture -0.226 -0.300 -0.152
devel_hi -0.097 -0.159 -0.035
forest -0.495 -0.580 -0.413
prcp2 0.035 0.033 0.037
prcp30 0.025 0.017 0.032
prcp2.da -0.044 -0.046 -0.042
prcp30.da -0.087 -0.095 -0.080
airTemp.da 0.054 0.028 0.081
airTemp.impoundArea -0.062 -0.086 -0.038
airTemp.agriculture -0.016 -0.039 0.005
airTemp.forest -0.019 -0.043 0.005
airTemp.devel_hi -0.007 -0.025 0.012
airTemp.prcp2 0.022 0.019 0.024
airTemp.prcp30 -0.053 -0.057 -0.049
airTemp.prcp2.da -0.016 -0.019 -0.014
airTemp.prcp30.da -0.014 -0.018 -0.010

5.1.2 HUC8 Random Effects

Figure 5.2 shows the estimated mean and 95% credible region interval (CRI) for each random effect and HUC8. Table 5.2 lists the estimated mean and 95% CRI of each parameter averaged over all HUC8s (mean value with standard deviation in parentheses).

Estimated Mean and 95% CRI of HUC Random Effects for Each HUC8

Figure 5.2: Estimated Mean and 95% CRI of HUC Random Effects for Each HUC8

Table 5.2: Mean and 95% CRI of HUC8 Random Effects Averaged Over All HUC8s (Mean Value and Std. Dev. in Parentheses)
Variable Count Mean Lower CRI Upper CRI
intercept.huc8 138 0.000 (0.515) -0.789 (0.583) 0.786 (0.576)
airTemp 138 1.964 (0.207) 1.693 (0.253) 2.234 (0.221)
temp7p 138 1.428 (0.294) 1.064 (0.320) 1.792 (0.345)

5.1.3 Catchment Random Effects

Figure 5.3 shows the distribution of the estimated mean for each random effect term over all catchments. CRIs are not shown due to the large number of individual catchments (7389). Table 5.3 lists the estimated mean and 95% CRI of each parameter averaged over all catchments (mean value with standard deviation in parentheses).

Distribution of estimated mean for each random effect over all catchments

Figure 5.3: Distribution of estimated mean for each random effect over all catchments

Table 5.3: Estimated mean and 95% CRI for each random effect averaged over all catchments (mean value with std. dev. in parentheses)
Variable Count Mean Lower CRI Upper CRI
intercept.site 2,463 -0.000 (1.401) -0.804 (1.411) 0.803 (1.435)
airTemp 2,463 -0.000 (0.349) -0.303 (0.362) 0.304 (0.367)
temp7p 2,463 0.000 (0.340) -0.508 (0.399) 0.508 (0.355)

5.1.4 Year Random Effects

Figure 5.4 and Table 5.4 present the mean and 95% CRI of the intercept term for each year. Recall that there are no random effects for years other than the intercept.

Estimated Mean and 95% CRI of Intercept Random Effect for Each Year

Figure 5.4: Estimated Mean and 95% CRI of Intercept Random Effect for Each Year

Table 5.4: Estimated Mean and 95% CRI of Intercept Random Effect for Each Year
Year Mean Lower CRI Upper CRI
1991 -0.151 -0.513 0.176
1992 0.104 -0.205 0.435
1993 0.187 -0.122 0.533
1994 0.058 -0.229 0.349
1995 0.063 -0.194 0.330
1996 -0.140 -0.397 0.101
1997 0.138 -0.062 0.347
1998 0.056 -0.140 0.253
1999 0.102 -0.080 0.283
2000 -0.333 -0.462 -0.214
2001 0.009 -0.107 0.126
2002 -0.016 -0.127 0.094
2003 -0.184 -0.310 -0.069
2004 0.150 0.031 0.265
2005 0.115 0.002 0.234
2006 -0.141 -0.245 -0.038
2007 -0.179 -0.284 -0.080
2008 0.057 -0.050 0.158
2009 0.073 -0.029 0.174
2010 0.169 0.072 0.259
2011 -0.090 -0.179 -0.004
2012 0.165 0.076 0.250
2013 0.112 0.022 0.197
2014 -0.070 -0.160 0.012
2015 -0.231 -0.319 -0.145
2016 0.207 0.118 0.297
2017 -0.225 -0.326 -0.126

5.2 Goodness-of-Fit

5.2.1 All Observations

Table 5.5 summarizes the model goodness-of-fit for all observations in the calibration an dvalidation datasets. Values in parentheses exclude the temporal auto-correlation term from the prediction calculations and thus represent model performance for ungauged catchments or time periods when observation data are not available.

Table 5.5: Summary statistics of model calibration and validation (values in parentheses denote value when temporal auto-correlation term is excluded)
Calibration Validation
# Observations 546,051 60,715
# Time Series 6,442 679
# Catchments 2,463 465
# HUC8s 138 94
# Years 27 22
RMSE (degC) 0.623 (1.111) 0.668 (1.469)
Mean Residual (degC) 0.007 (0.062) 0.016 (0.120)
Median Residual (degC) 0.009 (0.077) 0.013 (0.114)
Mean Absolute Residual (degC) 0.464 (0.840) 0.495 (1.082)
Median Absolute Residual (degC) 0.359 (0.663) 0.383 (0.838)
Minimum Residual (degC) -8.463 (-10.395) -7.824 (-12.045)
1st Percentile Residual (degC) -1.602 (-2.809) -1.696 (-3.550)
99th Percentile Residual (degC) 1.592 (2.834) 1.744 (3.998)
Maximum Residual (degC) 7.542 (13.373) 6.916 (11.046)

Figure 5.5 presents scatterplots of predicted vs. observed daily mean temperature for the calibration and validation datasets. The black line is the 1:1 line of equality. The red line is a linear regression trend line.

Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets (including temporal auto-correlation term)

Figure 5.5: Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets (including temporal auto-correlation term)

Figure 5.6 also presents scatterplots of predicted vs. observed daily mean temperature for the calibration and validation datasets but excludes the temporal auto-correlation term. The black line is the 1:1 line of equality. The red line is a linear regression trend line.

Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets (excluding temporal auto-correlation term)

Figure 5.6: Predicted versus Observed Daily Mean Temperature (degC) for Calibration and Validation Datasets (excluding temporal auto-correlation term)

5.2.2 Deployments

Table 5.6 summarises the mean, median, minimum and maximum RMSE for each deployment (i.e. continuous timeseries of observations at a single location) in the calibration and validation datasets.

Table 5.6: Summary statistics of model calibration and validation RMSE for each deployment (values in parentheses exclude temporal autocorrelation term)
Calibration Validation
# Time Series 6442 679
Mean RMSE (degC) 0.615 (1.005) 0.658 (1.274)
Median RMSE (degC) 0.581 (0.899) 0.617 (1.071)
Minimum RMSE (degC) 0.120 (0.088) 0.201 (0.251)
Maximum RMSE (degC) 3.881 (6.846) 2.049 (7.441)

Figure 5.7 shows the distribution of deployment RMSE including the temporal autocorrelation term.

Distribution of deployment RMSE including temporal autocorrelation term

Figure 5.7: Distribution of deployment RMSE including temporal autocorrelation term

Figure 5.8 shows the distribution of deployment RMSE excluding the temporal autocorrelation term.

Distribution of deployment RMSE excluding temporal autocorrelation term

Figure 5.8: Distribution of deployment RMSE excluding temporal autocorrelation term

5.2.2.1 Calibration Deployment Examples

Figures 5.9 to 5.12 show example deployments from the calibration dataset with the highest and lowest RMSE and including or excluding the temporal autocorrelation term.

Deployments with lowest RMSE in calibration dataset and n >= 30 including temporal autocorrelation term

Figure 5.9: Deployments with lowest RMSE in calibration dataset and n >= 30 including temporal autocorrelation term

Deployments with highest RMSE in calibration dataset and n >= 30 including temporal autocorrelation term

Figure 5.10: Deployments with highest RMSE in calibration dataset and n >= 30 including temporal autocorrelation term

Deployments with lowest RMSE in calibration dataset and n >= 30 excluding temporal autocorrelation term

Figure 5.11: Deployments with lowest RMSE in calibration dataset and n >= 30 excluding temporal autocorrelation term

Deployments with highest RMSE in calibration dataset and n >= 30 excluding temporal autocorrelation term

Figure 5.12: Deployments with highest RMSE in calibration dataset and n >= 30 excluding temporal autocorrelation term

5.2.2.2 Validation Deployment Examples

Figures 5.13 to 5.16 show example deployments from the validation dataset with the highest and lowest RMSE and including or excluding the temporal autocorrelation term.

Deployments with lowest RMSE in validation dataset and n >= 30 including temporal autocorrelation term

Figure 5.13: Deployments with lowest RMSE in validation dataset and n >= 30 including temporal autocorrelation term

Deployments with highest RMSE in validation dataset and n >= 30 including temporal autocorrelation term

Figure 5.14: Deployments with highest RMSE in validation dataset and n >= 30 including temporal autocorrelation term

Deployments with lowest RMSE in validation dataset and n >= 30 excluding temporal autocorrelation term

Figure 5.15: Deployments with lowest RMSE in validation dataset and n >= 30 excluding temporal autocorrelation term

Deployments with highest RMSE in validation dataset and n >= 30 excluding temporal autocorrelation term

Figure 5.16: Deployments with highest RMSE in validation dataset and n >= 30 excluding temporal autocorrelation term

5.2.3 Catchments

Table 5.7 summarises the mean, median, minimum and maximum RMSE of all catchments in the calibration and validation datasets.

Table 5.7: Summary of catchment RMSE values for calibration and validation datasets (values in parentheses exclude temporal autocorrelation term)
Calibration Validation
# Time Series 2463 465
Mean RMSE (degC) 0.571 (0.942) 0.648 (1.338)
Median RMSE (degC) 0.554 (0.856) 0.616 (1.116)
Minimum RMSE (degC) 0.181 (0.225) 0.292 (0.389)
Maximum RMSE (degC) 1.700 (3.398) 2.049 (7.441)

Figure 5.17 shows the distribution of catchment RMSE including the temporal autocorrelation term.

Distribution of catchment RMSE including temporal autocorrelation term

Figure 5.17: Distribution of catchment RMSE including temporal autocorrelation term

Figure 5.18 shows the distribution of catchment RMSE excluding the temporal autocorrelation term.

Distribution of catchment RMSE excluding temporal autocorrelation term

Figure 5.18: Distribution of catchment RMSE excluding temporal autocorrelation term