Defining coverage of an operational domain using a modified nearest-neighbor metric

Sez Atamturktur, Matthew C. Egeberg, François M. Hemez, Garrison N. Stevens

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

Validation experiments are conducted at discrete settings within an operational domain to assess the predictive maturity of a model that is ultimately used to predict over the entire operational domain. Unless this domain is sufficiently explored with validation experiments, satisfactory model performance at these discrete, tested settings would be insufficient to ensure satisfactory model performance throughout the entire operational domain. The goal of coverage metrics is then to reveal how well a set of validation experiments represents an operational domain. The authors identify the criteria of an exemplary coverage metric, evaluate the ability of existing coverage metrics to fulfill these criteria, and propose a new, improved coverage metric. The proposed metric favors interpolation over extrapolation through a penalty function, causing the metric to prefer a design of validation experiments near the boundaries of the domain, while simultaneously exploring inside the domain. Furthermore, the proposed metric allows the coverage to account for the relative influence of each dimension of the domain on the model output. Application of the proposed coverage metric on a practical, non-trivial two-dimensional problem is demonstrated on the Viscoplastic Self-Consistent material plasticity code for 5182 aluminum alloy. Furthermore, the proposed metric is compared to existing coverage metrics considering a high dimensional problem with application to the Rosenbrock function.

Original languageEnglish (US)
Pages (from-to)349-361
Number of pages13
JournalMechanical Systems and Signal Processing
Volume50-51
DOIs
StatePublished - Jan 2015

Fingerprint

Experiments
Extrapolation
Design of experiments
Plasticity
Aluminum alloys
Interpolation

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Signal Processing
  • Civil and Structural Engineering
  • Aerospace Engineering
  • Mechanical Engineering
  • Computer Science Applications

Cite this

Atamturktur, Sez ; Egeberg, Matthew C. ; Hemez, François M. ; Stevens, Garrison N. / Defining coverage of an operational domain using a modified nearest-neighbor metric. In: Mechanical Systems and Signal Processing. 2015 ; Vol. 50-51. pp. 349-361.
@article{790f00c85cee4f0a9aa88ea381e1e4d3,
title = "Defining coverage of an operational domain using a modified nearest-neighbor metric",
abstract = "Validation experiments are conducted at discrete settings within an operational domain to assess the predictive maturity of a model that is ultimately used to predict over the entire operational domain. Unless this domain is sufficiently explored with validation experiments, satisfactory model performance at these discrete, tested settings would be insufficient to ensure satisfactory model performance throughout the entire operational domain. The goal of coverage metrics is then to reveal how well a set of validation experiments represents an operational domain. The authors identify the criteria of an exemplary coverage metric, evaluate the ability of existing coverage metrics to fulfill these criteria, and propose a new, improved coverage metric. The proposed metric favors interpolation over extrapolation through a penalty function, causing the metric to prefer a design of validation experiments near the boundaries of the domain, while simultaneously exploring inside the domain. Furthermore, the proposed metric allows the coverage to account for the relative influence of each dimension of the domain on the model output. Application of the proposed coverage metric on a practical, non-trivial two-dimensional problem is demonstrated on the Viscoplastic Self-Consistent material plasticity code for 5182 aluminum alloy. Furthermore, the proposed metric is compared to existing coverage metrics considering a high dimensional problem with application to the Rosenbrock function.",
author = "Sez Atamturktur and Egeberg, {Matthew C.} and Hemez, {Fran{\cc}ois M.} and Stevens, {Garrison N.}",
year = "2015",
month = "1",
doi = "10.1016/j.ymssp.2014.05.040",
language = "English (US)",
volume = "50-51",
pages = "349--361",
journal = "Mechanical Systems and Signal Processing",
issn = "0888-3270",
publisher = "Academic Press Inc.",

}

Defining coverage of an operational domain using a modified nearest-neighbor metric. / Atamturktur, Sez; Egeberg, Matthew C.; Hemez, François M.; Stevens, Garrison N.

In: Mechanical Systems and Signal Processing, Vol. 50-51, 01.2015, p. 349-361.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Defining coverage of an operational domain using a modified nearest-neighbor metric

AU - Atamturktur, Sez

AU - Egeberg, Matthew C.

AU - Hemez, François M.

AU - Stevens, Garrison N.

PY - 2015/1

Y1 - 2015/1

N2 - Validation experiments are conducted at discrete settings within an operational domain to assess the predictive maturity of a model that is ultimately used to predict over the entire operational domain. Unless this domain is sufficiently explored with validation experiments, satisfactory model performance at these discrete, tested settings would be insufficient to ensure satisfactory model performance throughout the entire operational domain. The goal of coverage metrics is then to reveal how well a set of validation experiments represents an operational domain. The authors identify the criteria of an exemplary coverage metric, evaluate the ability of existing coverage metrics to fulfill these criteria, and propose a new, improved coverage metric. The proposed metric favors interpolation over extrapolation through a penalty function, causing the metric to prefer a design of validation experiments near the boundaries of the domain, while simultaneously exploring inside the domain. Furthermore, the proposed metric allows the coverage to account for the relative influence of each dimension of the domain on the model output. Application of the proposed coverage metric on a practical, non-trivial two-dimensional problem is demonstrated on the Viscoplastic Self-Consistent material plasticity code for 5182 aluminum alloy. Furthermore, the proposed metric is compared to existing coverage metrics considering a high dimensional problem with application to the Rosenbrock function.

AB - Validation experiments are conducted at discrete settings within an operational domain to assess the predictive maturity of a model that is ultimately used to predict over the entire operational domain. Unless this domain is sufficiently explored with validation experiments, satisfactory model performance at these discrete, tested settings would be insufficient to ensure satisfactory model performance throughout the entire operational domain. The goal of coverage metrics is then to reveal how well a set of validation experiments represents an operational domain. The authors identify the criteria of an exemplary coverage metric, evaluate the ability of existing coverage metrics to fulfill these criteria, and propose a new, improved coverage metric. The proposed metric favors interpolation over extrapolation through a penalty function, causing the metric to prefer a design of validation experiments near the boundaries of the domain, while simultaneously exploring inside the domain. Furthermore, the proposed metric allows the coverage to account for the relative influence of each dimension of the domain on the model output. Application of the proposed coverage metric on a practical, non-trivial two-dimensional problem is demonstrated on the Viscoplastic Self-Consistent material plasticity code for 5182 aluminum alloy. Furthermore, the proposed metric is compared to existing coverage metrics considering a high dimensional problem with application to the Rosenbrock function.

UR - http://www.scopus.com/inward/record.url?scp=84905814430&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84905814430&partnerID=8YFLogxK

U2 - 10.1016/j.ymssp.2014.05.040

DO - 10.1016/j.ymssp.2014.05.040

M3 - Article

AN - SCOPUS:84905814430

VL - 50-51

SP - 349

EP - 361

JO - Mechanical Systems and Signal Processing

JF - Mechanical Systems and Signal Processing

SN - 0888-3270

ER -