src.blocks.Assessment module

This module is responsible for assessing results from simulations and experiments.

It includes one class for the assessment. See details in its own documentation.

Contact person: Stefan Riedmaier Creation date: 20.04.2020 Python version: 3.8

class src.blocks.Assessment.Assessment(config, domain, instance)

Bases: object

This class is responsible for assessing results from simulations and experiments.

It includes a main method called “run_process” that calls the other class methods. See more details in the documentation of the run_process method.

calculate_kpis(qois_ts_da)

This function calculates the Key Performance Indicator (KPI) values.

It extracts characteristic values or KPIs from the time signals.

Available KPIs: - none: time signals of QoIs used without extraction of characteristic values - min: minimum value of a OoI signal - max: maximum value of a QoI signal - mean: mean value of a QoI signal

Parameters:

qois_ts_da (xr.DataArray) – time responses of the quantities of interest

Returns:

kpis of the quantities of interest (or time series if “none” in self.cfgas[‘qois_kpi_list’])

Rytpe:

xr.DataArray

calculate_qois(responses_da)

This function calculates the Quantity of Interest (QoI) signals.

The quantities from simulation or experiment can be selected directly as QoIs. It is also possible to combine mutliple response quantities to a new QoI. An example would be the calculation of a Time-to-Collision (TTC) signal from the distance and relative velocity signals of two vehicles.

Available QoIs: - Quantities from simulation and experiment - Time-To-Collision (TTC)

Parameters:

responses_da (xr.DataArray) – time responses of the original quantities

Returns:

time responses of the quantities of interest

Rytpe:

xr.DataArray

static calculate_ttc(responses)

This function calculates the Time-to-Collision (TTC) time signal.

The TTC is the ratio between the distance and the relative velocity between the ego and a traffic vehicle.

This function also corrects values from virtual CarMaker sensors, because CarMaker uses zero as default value if the traffic vehicle is out of the sensor range. However, zero can already have a meaning, such as an accident for TTC=0 or equal velocities for a relative velocity of zero. Thus, the default values are currently corrected to np.nan to avoid misleading overlaps.

Required quantities: - ‘AccelCtrl.ACC.Time2Collision’: directly the TTC signal from the simulator, or - ‘Sensor.Object.RadarL.relvTgt.NearPnt.ds.x’: distance between ego and traffic vehicle, and - ‘Sensor.Object.RadarL.relvTgt.NearPnt.dv.x’: relative velocity between ego and traffic vehicle

Parameters:

responses (xr.DataArray) – time responses from simulation or experiment

Returns:

array of TTC signals

Return type:

xr.DataArray

static create_ecdf(kpi_da)

This function creats empirical cumulative distribution functions (ECDFs).

It sorts the x-values and calculates the corresponding y-values / probabilities. The x values of the return array can be assessed via da.data. The y values of the return array can be assessed via da.probs.

It keeps duplicate values in the data array that get paired with successive probabilities. This is realized by using only sort instead of unique. It can be imagined as invisible substeps within one actual step. It has the advantage that - the probs correspond to the ‘linspace’ function and - the data and probs array can use their full shape without the need for complex masking or nan-filling. The functions that deal with ECDFs are prepared for this ECDF representation. This includes the Metric, Error Integration, and Decision Making classes as well as the ECDF step plotting. From a strict mathematical function point of view, only the actual last step counts.

The ECDF steps always refer to the ‘repetition’ or ‘aleatory_samples’ dimension. Theoretically, all ‘qois’, ‘space_samples’, or ‘epistemic_samples’ could have distinct repetitions and probs. However, in practice, only experiments with different repetitions for different space samples are of relevance. Each QoI and each epistemic sample has the same repetitions. Thus, we keep the probs array sparse and do not repeat it to the same shape as the data array.

Parameters:

kpi_da (xr.DataArray) – array with the processed kpis

Returns:

array with the processed kpis, converted to a ECDF representation

Return type:

xr.DataArray

static extract_scenario_timeseries(quantities_ts_da, scenarios_da)

This function extracts timeseries for scenario parameters from the measured quantity time signals.

Parameters:

  • quantities_ts_da (xr.DataArray) – data-array with time series of the quantities

  • scenarios_da (xr.DataArray) – data-array with scalar scenario parameters

Returns:

data-array with time series of the scenario parameters

Return type:

xr.DataArray

static get_pbox_edges(kpi_da)

This function extracts both edges of the ECDFs within the input array.

It replaces the ‘epistemic_samples’-dimension with a ‘pbox_edges’-dimension with a left and right label.

Parameters:

kpi_da (xr.DataArray) – array with the processed kpis, converted to a ECDF representation

Returns:

array with the processed kpis, converted to pbox edges representation

static perform_filtering(responses_da)

This function handles the filtering of the total data array including possible nan-values.

Unfortunately the scipy signal.filtfilt function (e.g. for butterworth) can not handle nan-values. If the input signal contains nan values, each element of the filtered signal is nan. However, our data array representation includes nan values to fill up values, since not all physical test scenarios have the same duration.

Parameters:

responses_da (xr.DataArray) – quantities data array

Returns:

filtered quantities data array

Return type:

xr.DataArray

run_process(quantities_ts_da, scenarios_kpi_da)

This function performs the assessment process of results from simulations and experiments.

It consists of multiple steps based on separate functions: 1) perform_filtering: filtering of noisy measurement quantities 2) calculate_qois: calculation of Quantities of Interest (QoIs) from the measured quantities 3) data-driven pipeline, partly implemented for the UNECE-R79 use case 3.1) extract_scenario_timeseries: calculation of scenario time series from measured quantities 3.2) unece_event_finder.find_lkft_events: data-driven event finding (trimming of scenario and qoi time series) 3.3) calculation of scenario parameters from scenario time series 3.4) cluster_lkft_events: clustering of events to determine repetitions, etc. 4) calculate_kpis: calculation of Key Performance Indicators (KPIs) from the qoi time signals 5) create_ecdf: creation of empirical CDFs from aleatory samples (and repetitions) 6) get_pbox_edges: creation of p-boxes from epistemic samples

The qoi array can have different dimensions depending on the selected case: - ‘qois’, ‘space_samples’: deterministic simulations with KPI assessment - ‘qois’, ‘space_samples’, ‘timesteps’: deterministic simulations without KPI assessment - ‘qois’, ‘space_samples’, ‘aleatory_samples’/’repetitions’: ECDFs based on KPI assessment - ‘qois’, ‘space_samples’, ‘pbox_edges’, ‘aleatory_samples’: p-boxes based on KPI assessment

Parameters:

  • quantities_ts_da (xr.DataArray) – array of quantity time series (ts)

  • scenarios_kpi_da (xr.DataArray) – array of scenario parameters (kpis)

Returns:

array of processed qois and scenario parameters

Return type:

tuple(xr.DataArray, xr.DataArray)