This paper explores the effectiveness of the recently devel- oped surrogate modeling method, the Adaptive Hybrid Functions (AHF), through its application to complex engineered systems design. The AHF is a hybrid surrogate modeling method that seeks to exploit the advantages of each component surrogate. In this paper, the AHF integrates three component surrogate mod- els: (i) the Radial Basis Functions (RBF), (ii) the Extended Ra- dial Basis Functions (E-RBF), and (iii) the Kriging model, by characterizing and evaluating the local measure of accuracy of each model. The AHF is applied to model complex engineer- ing systems and an economic system, namely: (i) wind farm de- sign; (ii) product family design (for universal electric motors); (iii) three-pane window design; and (iv) onshore wind farm cost estimation. We use three differing sampling techniques to inves- tigate their influence on the quality of the resulting surrogates. These sampling techniques are (i) Latin Hypercube Sampling
∗Doctoral Student, Multidisciplinary Design and Optimization Laboratory, Department of Mechanical, Aerospace and Nuclear Engineering, ASME student member.
†Distinguished Professor and Department Chair. Department of Mechanical and Aerospace Engineering, ASME Lifetime Fellow. Corresponding author.
‡Associate Professor, Department of Mechanical Aerospace and Nuclear En- gineering, ASME member (LHS), (ii) Sobol’s quasirandom sequence, and (iii) Hammers- ley Sequence Sampling (HSS). Cross-validation is used to evalu- ate the accuracy of the resulting surrogate models. As expected, the accuracy of the surrogate model was found to improve with increase in the sample size. We also observed that, the Sobol’s and the LHS sampling techniques performed better in the case of high-dimensional problems, whereas the HSS sampling tech- nique performed better in the case of low-dimensional problems. Overall, the AHF method was observed to provide acceptable- to-high accuracy in representing complex design systems.
1. Surrogate Modeling of Complex Systems Using
Adaptive Hybrid Functions
Jie Zhang*, Souma Chowdhury*, Achille Messac#
Junqiang Zhang* and Luciano Castillo*
* Rensselaer Polytechnic Institute, Department of Mechanical, Aerospace, and Nuclear Engineering
# Syracuse University, Department of Mechanical and Aerospace Engineering
ASME 2011 International Design Engineering Technical Conferences (IDETC)
and Computers and Information in Engineering Conference (CIE)
37th Design Automation Conference (DAC)
August 28 – 31, 2011
Washington, DC
2. Introductory Observation
• The need to quantify complex system performance often demands
computationally expensive simulations and/or expensive experiments.
• Surrogate modeling provides approximation models to represent the
relationships between specific system inputs and outputs, which can be
used to estimate the system performance for any given input.
• The hybrid surrogate modeling paradigm, which seeks to combine
attractive features of different surrogates, offers a promising approach
towards development of high fidelity approximation models.
Hybrid
Surrogate
Model
Kriging RBF E-RBF 2
3. Applications
3
Art
Chemistry
Math Automotive
Biology
Geology Data mining Material Science
Source: Google Images
4. Research Objectives
4
This paper explores the wide applicability of the recently
developed hybrid surrogate: Adaptive Hybrid Functions
(AHF).
Previous paper established effectiveness of AHF versus
individual surrogates.
Apply AHF to complex engineered systems design, and
economic system design problems.
This paper implements three representative sampling
techniques (i) Latin Hypercube Sampling (LHS), (ii)
Sobol’s quasirandom sequence, and (iii) Hammersley
Sequence Sampling (HSS).
Investigate the effects of sample size and problem
dimensionality on the performance of the surrogate model.
5. Outline
• Surrogate Modeling Review
• Adaptive Hybrid Functions (AHF)
• Complex Engineered and Economic Systems
• Wind Farm Design
• Product Family Design (for Universal Electric Motors)
• Three-Pane Window Design
• Onshore Wind Farm Cost Model
• Results and Discussion
5
6. Surrogate Modeling Review
Parametric & Nonparametric Surrogate Modeling
6
Hybrid Surrogate Models
Weighted averaged surrogates1
Ensemble of surrogates using Generalized Mean Square Cross-validation
Error2
Optimization on the weights3
Based on various local error measures4
Using recursive process to obtain the weights5
1Zepra et al. 2 Goel et al. 3 Acar and Rais-Rohani 4Acar 5Zhou et al.
8. AHF Framework
Step A.1: Determination Step A of the Base Model
Determination of a trust region:
Numerical bounds of the estimated
parameter (output) as functions of
the input vector over the feasible
space.
Characterization of the local
measure of accuracy: Using kernel
functions of the estimated output
value.
8
9. 9
Step A.2: Formulation of Crowding Distance-Based Trust Region (CD-TR)
Wherever the density of training points is high, the interpolative function in that region is
expected to have smaller errors.
Crowding distance is used to evaluate the
density of points:
A parameter ρ is defined to represent the local
density of input data:
The adaptive distance:
AHF Framework
It is important to note that the CD-TR estimation is particularly useful for data obtained from
experiments/simulation that was not preceded by Design of Experiments. In the case of problems,
where the user has control over sampling, the initial sample data is expected to be relatively evenly
distributed; significant variation in crowding distance is unlikely.
10. 10
AHF Framework
Accuracy Measure of Surrogate Modeling (AMSM)
We develop an Accuracy Measure of Surrogate
Modeling, to represent the uncertainty in the
estimated function value.
The kernel function provides a measure of the
accuracy of component surrogates.
The corresponding coefficients of the kernel
function are represented as functions of the input
vector, thereby characterizing the measure of
accuracy of the estimated function over the entire
input domain.
The following kernel function is adopted here
11. AHF Framework
11
We assume that the reliability of the estimated measure of accuracy (kernel function) is
a maximum of one at the actual output value y(xi); and a minimum of 0.1 at the trust
region boundaries. The kernel function is represented as
σ1 and σ2 are controlled by the full width at one tenth maximum (Δz10), given by
and
where
12. Step B: Component Surrogates Development
In this paper, the AHF integrates:
Kriging method
Radial Basis Functions (RBF)
Extended Radial Basis Functions (E-RBF)
12
13. Step C: Determining Local Weights
13
The AHF is a weighted summation of function values estimated by the component
surrogates:
The weights are expressed in terms of the estimated measure of accuracy, expressed as
where, Pi(x) is the measure of accuracy of the ith surrogate for point x.
14. Complex Engineered and Economic Systems
The AHF is applied to complex engineered design problems
and an economic system:
1) Wind Farm Design
2) Product Family Design (for Universal Electric Motors)
3) Three-Pane Window Design
4) Onshore Wind Farm Cost Model
14
Problems Challenges to Surrogate
Modeling
Computational
Cost
Wind Farm Design Highly nonlinear, multimodal Low
Product Family Nonlinear, multimodal Fair
Three-pane Window Design Highly nonlinear High
Wind Farm Cost Model No Design of Experiment Commercial data
15. Wind Farm Design
15
The power generated by a wind farm
The farm efficiency
We develop a hybrid response surface (using the AHF) to represent the farm efficiency as a
function of the turbine location coordinates.
We consider four cases:
wind farm with 4 turbines (8 variables);
wind farm with 9 turbines (18 variables);
wind farm with 16 turbines (32 variables); and
wind farm with 25 turbines (50 variables).
Challenges to surrogate modeling:
Highly nonlinear (wake model, wake overlap, power generation model)
Multimodal (power generation model)
16. Product Family Design
16
Comprehensive Product Platform Planning (CP3) framework
AHF method is used to represent the two objectives and the two constraints as functions of
design variables:
Objectives: performance objective (fperf) and cost objective (fcost)
Constraints: system constraint and commonality constraint
We consider three cases:
2 products (21 variables);
3 products (28 variables); and
4 products (35 variables).
Challenges to surrogate modeling:
The performance function and the system
constraint are fairly nonlinear.
The commonality constraint is nonlinear and
particularly multimodal.
Design variable limits of the electric motors
17. Three-Pane Window Design
The heat transfer simulation model of the side channels and the air gap is created using
the computational fluid dynamics (CFD) software Fluent.
The inputs for the surrogate model are
17
The atmospheric temperature,
The wind speed, and
The solar radiation.
The output of the surrogate model is
The heat flux through the inner pane, Qwindow.
Challenges to surrogate modeling:
Highly nonlinear (CFD model).
Computational expensive.
18. Onshore Wind Farm Cost Model
18
Response Surface-Based Wind Farm Cost (RS-WFC) model
The inputs for the surrogate model are
The number, and
The rated power of wind turbines.
The output of the surrogate model is
Total annual cost of a wind farm
Challenges to surrogate modeling:
Not preceded by Design of Experiment.
19. Performance Criteria
19
Root Mean Squared Error (RMSE): overall performance
Maximum Absolute Error (MAE): maximum deviation
Relative Accuracy Error (RAE)
Cross-Validation: (i) leave-one-out strategy, and (ii) q-fold strategy
20. Experimental Designs
20
We consider three sampling methods:
Latin Hypercube Sampling (LHS)
The values of the np numbers of points in each column are randomly
selected - one from each of the intervals, (0, 1/np); (1/np, 2/np), ..., (1-
1/np, 1).
Sobol’s Quasirandom Sequence Generator
Sobol sequences use a base of two to form finer uniform partitions of
the unit interval, and reorder the coordinates in each dimension.
Hammersley Sequence Sampling (HSS)
The HSS is based on the representation of a decimal number in the
inverse radix format, where the radix values are chosen as the first
(m-1) prime numbers, m being the number of dimensions.
22. Results and Discussion
In the case of the three-pane window, the HSS technique performs better than the LHS
22
Effect of Sampling Technique and Size
Three-pane window model
and Sobol techniques;
For the wind farm power generation model, the LHS technique performs better than the
other two sampling methods;
For the product family design problem, in terms of the RMSE values, both LHS and
Sobol provide better performance. However, the HSS yields smaller MAE values.
WindP rpoodwucetr fgaemnielrya tmioond melodel
The errors (RMSE, MAE, PRESS) do not consistently decrease with
increasing sample size, likely owing to training point sensitivity.
23. Effect of Problem Dimensionality
23
In the case of Sobol and LHS sampling techniques, the values of
RMSE, MAE, and PRESS decrease when the dimension increases;
In the case of HSS sampling technique, the AHF method has high
accuracy for relatively lower dimensional problems.
24. Conclusion
• This paper presented applications of the Adaptive Hybrid Functions
(AHF) to represent complex engineered and economic systems.
• The errors (RMSE, MAE, PRESS) do not consistently decrease with
increasing sample size, likely owing to training point sensitivity.
• The application of AHF using Sobol’s and LHS sampling provides
relatively better accuracy for high dimensional problems.
• The application of AHF using HSS sampling provides relatively better
accuracy for low dimensional problems.
• Future research should investigate adaptive sampling strategies to
provide a more realistic coverage of the design domain for surrogate
model development.
24
25. Acknowledgement
• I would like to acknowledge my research adviser
Prof. Achille Messac, and my co-adviser Prof.
Luciano Castillo for their immense help and
support in this research.
• I would also like to thank my friends and colleagues
Souma Chowdhury and Junqiang Zhang for their
valuable contributions to this paper.
• I would also like to thank NSF for supporting this
research.
25
27. Selected References
1. Goel, T., Haftka, R., Shyy, W., and Queipo, N., "Ensemble of Surrogates," Structural and Multidisciplinary
Optimization, Vol. 33, No. 3, 2007, pp. 199-216.
2. Acar, E. and Rais-Rohani, M., "Ensemble of Metamodels with Optimized Weight Factors," Structural and
Multidisciplinary Optimization, Vol. 37, No. 3, 2009, pp. 279-294.
3. Viana, F., Haftka, R., and Steffen, V., "Multiple Surrogates: How Cross-validation Errors Can Help Us to Obtain the
Best Predictor," Structural and Multidisciplinary Optimization, Vol. 39, No. 4, 2009, pp. 439-457.
4. Forrester, A., Sobester, A., and Keane, A., Engineering Design via Surrogate Modelling: A Practical Guide, Wiley,
2008.
5. Simpson, T., A Concept Exploration Method for Product Family Design, Ph.D. thesis, Georgia Institute of Technology,
1998.
6. Jin, R., Chen, W., and Simpson, T., "Comparative Studies of Metamodelling Techniques Under Multiple Modelling
Criteria," Structural and Multidisciplinary Optimization, Vol. 23, No. 1, 2001, pp. 1-13.
7. Mullur, A. and Messac, A., "Extended Radial Basis Functions: More Flexible and Effective Metamodeling," AIAA
Journal, Vol. 43, No. 6, 2005, pp. 1306-1315.
8. Mullur, A. and Messac, A., "Extended Radial Basis Functions: More Flexible and Effective Metamodeling," AIAA
Journal, Vol. 43, No. 6, 2005, pp. 1306-1315.
9. Queipo, N., Haftka, R., Shyy, W., Goel, T., Vaidyanathan, R., and Tucker, P., "Surrogate-based Analysis and
Optimization," Progress in Aerospace Sciences, Vol. 41, No. 1, 2005, pp. 1-28.
10. Wang, G. and Shan, S., "Review of Metamodeling Techniques in Support of Engineering Design Optimization,"
Journal of Mechanical Design, Vol. 129, No. 4, 2007, pp. 370-380.
27
28. 28
Test Function 1: 1-Variable Function
Test Function 2: 2-Variable Function
Test Function 3: Goldstein & Price Function
Test Function 4: Branin-Hoo Function
Test Function 5 and 6: Hartmann Function
29. Kriging
29
The kriging approximation function consists of two parts: (i) a
global trend function, and (ii) a functional departure from the trend
function.
In this paper, we use a Matlab Kriging
toolbox DACE (Design and Analysis
of Computer Experiments), developed
by Dr. Nielsen.
30. Radial Basis Functions
30
Radial Basis Functions
The RBFs are expressed in terms of the Euclidean distance,
y (r) = r2 + c2
where c > 0 is a prescribed parameter.
( )
% =å -
f x sy x x
1
( )
np
i
i
i
=
r = x - xi
One of the most effective forms is the multiquadric function:
The final approximation function is a linear combination of these basis
functions across all data points.
31. Extended Radial Basis Function (E-RBF)
31
Extended Radial Basis Functions (E-RBF) is a combination of Radial
Basis Functions (RBFs) and Non-Radial Basis Functions (N-RBFs).
Non-Radial Basis Functions
N-RBFs are functions of individual coordinates of generic points x
relative to a given data point xi, in each dimension separately
It is composed of three distinct components
E-RBF
The E-RBF approach incorporates both the RBFs and the N-RBFs
Methods: (i) linear programming, or (ii) pseudo inverse.
32. 32
Method Parameter value
E-RBF λ = 4.75; c = 0.9; t = 2
RBF c = 0.9
Kriging θl = 0.1; θu = 20
Cross-validation q = 5
33. Hammersley Sequence Sampling
The two-dimensional Hammersley point set of order is defined by taking all
numbers in the range from 0 to 2m-1 and interpreting them as binary fractions.
Calling these numbers xi, then the corresponding yi are obtained by reversing the
binary digits of xi
33
Hinweis der Redaktion
Overall, the presentation needs to be better paced. There is too much information and time spent on AHF.
This paper is about performance of AHF, and exploration of sampling strategies, sampling size, and dimensionality in the case of complex problems. You do not emphasize that enough in the introduction. The introduction makes this paper look like a new surrogate modeling paper. Please make corrections accordingly. My text changes are in green. Carefully check the notes at the bottom of each section for further comments.
Optional: You can show an animation of three different surrogates (side by side) , e.g. kriging, rbf, and qrsm which then move towards each other combine, and then the hybrid surrogate appears.
Give references at the bottom in small grey fonts for the hybrid surrogates
and the representation of the corresponding kernel function parameters as functions of the input vector.
Is “y(x)” normalized. If so, by what?
In a presentation, you do not need to say “as given by”, just use “:”
You still have RBHF in the graph. It should be AHF
I think you are spending too much time on the AHF, which you have already presented in SDM 2011
Spend more time in explaining the complexity of the 4 real life test cases, and how AHF addresses the challenging complexities.
Before this slide, Use one slide with a table: 4 test problems. In the next column, their challenges. In the next column, their computational expense.
Before this slide, Use one slide with a table: 4 test problems. In the next column, their challenges. In the next column, their computational expense.
Where are the features of these two problems. And why on the same page?
You are comparing sampling methods. You should atleast provide one separate slide with one/two line description of each sampling method. If you comparing them, and people do not even know their characteristic differences, you are not really teaching them anything.
It should be “Variable limits”
You are comparing sampling methods. You should atleast provide one separate slide with one/two line description of each sampling method. If you comparing them, and people do not even know their characteristic differences, you are not really teaching them anything.
It should be “Variable limits”
Underline “Three Pane Window Model” caption
Please comment why the errors do not decrease consistently with increasing sample size: most likely because, AHF is highly sensitive to the actual training points chosen. Everytime you generate a higher sample size data, the random training pts are mostly different (that may be one of the reasons).
The first observation is weird. Please explain.
Does the sample size proportionally scale up with the dimensionality.
Discuss each error separately. May be bring each figure separately and as a big figure, instead of all three together side by side.
Your second conclusion was incorrect. I have replaced it.
The last conclusion has no apparent significance. Also replaced.
It would also be very interesting to see, how the contribution of the different component surrogates change when problem dimensionality changes, and/or when sample size changes. The journal version of this topic should definitely include that.