1   Overview¶

The lsst.verify package provides a framework for characterizing the LSST Science Pipelines through specific metrics, which are configured by the verify_metrics package. However, lsst.verify does not specify how to organize the code that measures and stores the metrics. This document proposes a new framework that interacts with the lsst.verify and Task frameworks to make it easy to write new metrics and apply them to the Science Pipelines.

The proposed design, shown below, is similar to the make measurements from output datasets option proposed in DMTN-057. Each metric will have an associated lsst.pipe.base.Task class that is responsible for measuring it based on data previously written to a Butler repository. These tasks will be grouped together for execution, first as plugins to a central metrics-managing task, and later as components of a lsst.pipe.base.Pipeline. The central task or pipeline will handle the details of directing the potentially large number of input datasets to the measurement tasks that analyze them.

This design proposal strives to be consistent with the recommendations on metrics generation and usage provided by DMTN-085, without assuming them more than necessary. However, the design does require that QAWG-REC-34 (“Metric values should have Butler dataIds”) be adopted; otherwise, the output dataset type of the proposed measurement task would be ill-defined.

Figure 1 Classes introduced in this design (blue), including a compatibility module for adapting the framework to a non-PipelineTask system. For simplicity, Task config classes are treated as part of the corresponding Task.

2   Design Goals¶

The goals of this design are based on those presented in DMTN-057. In particular, the system must be easy to extend, must support a variety of metrics, and must be agnostic to how data processing is divided among task instances. It must not add extra configuration requirements to task users who are not interested in metrics, and it must be possible to disable metrics selectively during commissioning and operations.

DMTN-085 makes several recommendations that, if adopted, would impose additional requirements on a measurement creation framework. Specifically:

• QAWG-REC-31 recommends that the computation and aggregation of measurements be separated.
• QAWG-REC-32 recommends that measurements be stored at the finest granularity at which they can be reasonably defined.
• QAWG-REC-34 and QAWG-REC-35 recommend that measurements be Butler datasets with data IDs.
• QAWG-REC-41 recommends that metrics can be submitted to SQuaSH (and, presumably, measured first) from arbitrary execution environments

DMTN-085 Section 4.11.1 also informally proposes that the granularity of a metric be part of its definition, and notes that some metrics may need to be measured during data processing pipeline execution, rather than as a separate step. I note in the appropriate sections how these capabilities can be accommodated.

The capabilities and requirements of PipelineTask are much clearer now than they were when DMTN-057 was written (note: it refers to PipelineTask by its previous name, SuperTask). Since reusability outside the Tasks framework is not a concern, the design proposed here can be tightly coupled to the existing lsst.pipe.base.PipelineTask API.

3   Primary Components¶

The framework creates lsst.pipe.base.PipelineTask subclasses responsible for measuring metrics and constructing lsst.verify.Measurement objects. Metrics-measuring tasks (hereafter MetricTasks) will be added to data processing pipelines, and the PipelineTask framework will be responsible for scheduling metrics computation and collecting the results. It is expected that PipelineTask will provide some mechanism for grouping tasks together (e.g., sub-pipelines), which will make it easier to enable and configure large groups of metrics.

Because MetricTasks are handled separately from data processing tasks, the latter can be run without needing to know about or configure metrics. Metrics that must be calculated while the pipeline is running may be integrated into pipeline tasks as subtasks, with the measurement(s) being added to the list of pipeline task outputs, but doing so greatly reduces the flexibility of the framework and is not recommended.

PipelineTask is not available for general use at the time of writing, so initial implementations may need to avoid referring to its API directly. Any such substitutions will be marked in square brackets and the word “initially”. If a practical adapter cannot be developed (see 4   Compatibility Components), these substitutions may be implemented as differently-named methods in order to simultaneously support PipelineTask and non-PipelineTask execution in the same class.

For illustration all MetricTask classes are shown as members of a single verify_measurements package. However, this is not required by the framework; subclasses of MetricTask may be defined in the packages of the task they instrument, or in plugin packages similar to meas_extensions_*. The framework is therefore compatible with any future policy decisions concerning metric implementations.

4   Compatibility Components¶

We expect to deploy new metrics before PipelineTask is ready for general use. Therefore, the initial framework will include extra classes that allow MetricTask to function without PipelineTask features.

By far the best way to simultaneously deal with the incompatible Butler 2 and Butler 3 APIs would be an adapter class that allows MetricTask classes initially written without PipelineTask support to serve as lsst.pipe.base.PipelineTask. Unfortunately, the design of such an adapter is complicated by the strict requirements on PipelineTask constructor signatures and the use of configs as a Task‘s primary API.

I suspect that both problems may be solved by applying a decorator to the appropriate type objects rather than using a conventional class or object adapter for Task or Config objects, but the design of such an decorator is best addressed separately.