Differences

This shows you the differences between two versions of the page.

--- concepts:calibration_demystified [2009/10/08 18:38]
deryn.crockett
+++ concepts:calibration_demystified [2009/10/16 03:09]
marcus.williams
@@ Line 1: / Line 1: @@
 ====== Calibration Demystified ======
-Calibrating a simulation model involves calculating a set of input-variable values for a historical time period which reproduces - using the simulator framework - the observed simulator outputs over that same period. To accomplish this a formal calibrator framework is created alongside the simulator framework within a model family (see [[concepts:How it's Organized]]).
+Calibration is the process of finding simulator input values for a historical time period which reproduce the observed simulator outputs over that period. To accomplish this a distinct calibrator framework is created alongside the simulator framework, within the same model family (see [[concepts:How it's Organized]]). There are two main motivations for calibration. The first is to look back at the history of a system through the same conceptual framework used to explore the future of that system - a prerequisite for trend analysis. The second is to get starting values - or initial conditions - of model stocks.
-[Perhaps give a trivial example where the simulator model is //y = x + z//. If the historical observed value for y is 6, then a calibration solution for (x,y) could be (3,3), (2,4) or (1,5)...and so on. For real-world models this is a non-trivial problem and requires a separate calibrator framework because:
+Take for example a simple population model in which
-  * Simulators are dynamic and data-rich; they contain many inter-related, multidimensional time-series variables which constrain the calibration solution
+population<sub>t</sub> = population<sub>t-1</sub> + net immigration<sub>t-1</sub> + births<sub>t-1</sub> - deaths<sub>t-1</sub>
-  * Simulator output data may not be directly measured and therefore may have to be estimated, and/or there may be missing periods of observed data which require interpolation.
-  * [more?]
-] FIXME
+If historical data for population, births and deaths are available then calibration involves determining the historical net immigration levels.  For real-world models this is a non-trivial problem and requires a separate calibrator framework because:
+  * Simulators are dynamic and data-rich; they contain many inter-related, multidimensional time-series variables which constrain the calibration solution. In the example above the population variable, rather than being a vector, might be a 3-dimensional variable stratified by sex, age and time.
+  * Simulator variables in historical time may not be directly measured and therefore may have to be estimated, and/or there may be missing periods of observed data which require interpolation. For the population model example it is common to use a simulation time-step of 1 year, and while births and deaths data may be available for each year, population census count data is more likely to be collected every 5 or 10 years. The calibrator must deal with the varying temporal resolution of the datasets.
+  * Data quality issues are common, and often manifest themselves through inconsistent or infeasible calibration results. For example, suppose that net immigration in the above example were broken into its component parts - immigration and emigration. If estimates for historical immigration were available and used in the calibration to estimate emigration levels, it is possible that a naive calibration procedure would produce some negative values for historical emigration. Of course, this is infeasible and so the calibration framework must consider these issues to produce an internally consistent and feasible calibration solution.
+  * [more?]
-[Explain the diagram below wrt the frameworks, relative to time, inputs and outputs.  Break observed historical data btwn observed parameters and observed outputs.] FIXME
+The diagram below shows the relationship between a simulator framework and its corresponding calibrator framework - with respect to inputs, outputs and the time periods for which they operate. Key points to note:
+  * The simulator contains logic that transforms input variables to output variables, and this same logic is applied for future simulation (pink time period) and calibration (blue time period).
+  * A major difference between the simulator being run for future simulation vs. historical time has to do with the source of the inputs. For future simulation the inputs originate from the model users. Difference input sets give rise to different scenarios. For historical time, however, the simulator inputs are produced by the calibrator framework, whose job it is to ensure that simulator outputs match observed historical data. The calibrator logic is only exercised over historical time.
+  * "Observed historical data", or the inputs to the calibrator, can be further subdivided between observed outputs and paramters. [Need to find a more general and accurate term for Observed historical data and replace it in the diagram].
 {{:concepts:calibration.png|}}

whatIf? Wiki

User Tools

Site Tools

Differences

Page Tools