Modeling & Simulation of a fast reactor fuel assembly using commercial CFD software and high performance computers.

1. Parallel CFD Simulation
“Fast Reactor Assembly”
Kurt D. Hamman
Multi-Physics Methods Group
Nuclear Science & Energy
July 21, 2008

2. Presentation Overview
• Project Objectives
– Problem Size
• M&S Process
– “Pre-processing”
– “Solver Settings”
– “Post-processing”
• Validation Process
• HPC “Lesson’s Learned”
• Conclusions

3. Project Objectives
• Primary
– Develop a CFD Modeling Process using Commercial
Software for “large” problems
( CAD → Meshing → Simulation → Visualization )
• Secondary
– Evaluate Commercial Software
• CAD and CFD
– Evaluate HPC and Network Infrastructure
• Problem Size (“production-type” simulations)

4. Project Objectives – “Problem Size”
65 – 85 million elements
Meshing ~ 1 to 2 days
1 million ≈ 1 GB memory
File Size → ~ 25 GB HPC (Meshing)
Aurora (SGI Altix – 4700)
• Shared Memory - 256 GB
• 96 - 1.5 GHz Itanium 2 Processors
• SUSE Linux Enterprise Server 10
• 5.72 TB Disk Capacity
Simulation ~ 2 to 10 days
HPC (Simulation)
• Icestorm (Altix ICE 8200)
• Distributed Memory(16 GB/Node)
• 256 Nodes/8 PPN
• 2.66 GHz clock speed
• Linux OS

5. Modeling Process
19-Pin Fast Reactor Assembly
Geometry
&
Results
“Surface Mesh” Mesh
Meshing Solver +
CAD (HPC) (HPC) Validation
19 Pin Fuel Assembly
0.18 0.03500
0.16
0.03000
0.14
0.02500
0.12
0.02000
0.10
( kpa/cm )
Empirical
( cm )
Numerical
Pin Gap
0.08
0.01500
0.06
0.01000
0.04
Pin Gap 0.00500
0.02
0.00 0.00000
C(1) C(2) C(3)
Model
Validation
Results
Polyhedral Mesh
(65.5 million elements)
Solid Model

6. “Preprocessing” – CAD Modeling
• Geometry “similar” to Advanced Burner Test Reactor (ABTR)
19 Pin Assembly
Overlap
(0.0065 cm)
Todreas, N.E. & Kazimi, M.S., Nuclear Systems II
Elements of Thermal Hydraulic Design, Taylor and Francis, 2001.
Ds (3) Dft Dl
Model Pins D P/D P ΔP Δg Length T (2) F (2)
ABTR 217 0.800 0.103 1.130 0.904 0.001 0.033 13.598 - 260 0.0797 0.174
M(1) 19 0.800 0.103 1.169 0.936 0.039 0.032 4.299 2.482 20.0 0.2001 0.675
M(2) 19 0.800 0.103 1.149 0.919 0.023 0.016 4.210 2.431 20.0 0.1113 0.708
M(3) 19 0.800 0.103 1.135 0.908 0.012 0.005 4.148 2.395 20.0 0.0498 0.799
M(4) 19 0.800 0.103 1.127 0.902 0.005 0.005 4.126 2.382 20.3 0.0273 0.634
Notes:
1. All dimensions in centimeters.
2. Assembly geometry based on Todreas and Kazimi, where quot;Tquot; represents the flat-to-flat tolerance and quot;Fquot; represents the fraction of assembly tolerance distributed within the rods.

7. “Preprocessing” – Mesh
Boundary Layer
(Prism Cells)
Interior Volume
(Polyhedral Cells)

8. Modeling Process “Solver”
• 3 Dimensional
• “Steady-State”
• Sodium Properties (700 K)
• Constant Density
• Segregated Solver
• Second Order (convective)
• Turbulent Flow
• Turbulence Models: k-ε k-ω
• Boundary Conditions
– Inlet velocity 2 m/s
– Outlet pressure 0 psi
– Walls “no slip”
– Heat Flux ~ 1MW/m2

9. Modeling Process “Post-processing”

10. Modeling Process “Post-processing”

11. Modeling Process “Post-processing”

12. Validation Process
Examine Iterative Convergence

● 3 to 4 orders of magnitude
Examine Conservation

● conservation of mass/energy
X Examine Spatial (Grid) Convergence
● ordered discretization error
X Examine Temporal Convergence
● N/A → Steady State Simulation
Compare CFD Results to Data

● “Average” Pressure Drop
● “Average” Mass Flowrate
Examine Model Uncertainties

● Turbulence Models/Solvers
 Performed
X Not Performed
Reference: NPARC Alliance CFD Verification and Validation Web Site
(www.grc.nasa.gov/WWW/wind/valid/validation.html)

13. Validation – Iterative Convergence
Energy Equation Residual Response
1. Activating heat transfer (q” ~ 1 MW/m2 )
2. Sdr and Tke “noise
85 Million Element Model (3.02.003)

14. Validation – Examine Consistency
Polyhedral Mesh
Numerical
Interior Pin Wall-Pin Inlet Outlet Delta Difference
Clearance Gap mass flowrate mass flowrate Δm (wrt inlet)
Model (cm) (cm) (kg/s) (kg/s) (kg/s)
C(1) 0.039 0.032 1.083 1.083 6.000E-06 0.00%
C(2) 0.023 0.016 0.971 0.971 1.500E-06 0.00%
C(3) 0.012 0.005 0.896 0.896 6.900E-06 0.00%
Numerical
Interior Pin Wall-Pin Heat Transfer Heat Transfer Delta Difference
Clearance Gap quot;inquot; quot;outquot; ΔQ (wrt inlet)
Model (cm) (cm) (W) (W) (W)
C(1) 0.039 0.032 1.621E+06 1.620E+06 -1.636E+03 -0.10%
C(2) 0.023 0.016 1.453E+06 1.451E+06 -1.534E+03 -0.11%
C(3) 0.012 0.005 1.340E+06 1.338E+06 -1.666E+03 -0.12%

15. Validation-Spatial Grid Convergence
65 – 85 million elements
1 million ≈ 1 Gb memory
File Size → ~ 25 GB
(HPC Lessons Learned)

16. Validation-Temporal Convergence
• Steady-State Simulation
– Not performed
• General Comments
– File size ~27.5 GB
– 1000 second transient
– 27.5 TB storage
• Suggests a Need
– Parallel I/O Reference 5: “Advanced Burner Test Reactor Pre-Conceptual Design Report”
– File Storage

17. Sensitivity Analyses Overview
• Analysis #1
– Assembly Geometry Changes
– Several Turbulence Models
– Empirical Correlations
• Analysis #2
– Wirewrap Geometry Changes
– One Turbulence Model
– Empirical Correlations
• Analysis #3
– Mesh element types
(polyhedral, trimmer)

18. HPC “Lessons Learned”
• Understand Capabilities
– Network
– HPC Machines
– Software
– Productivity
• Optimization
– Memory Access
• Mesh Refinement
• Post processing
• Unknowns
– File Size

19. Understand Capabilities – “Network”
PRIMARY FIRE WALL
Subnet #1 Subnet #2 Subnet #3
e
Business
Enclave HPC Research
Enclave Enclave
Note: For illustration purposes only.

20. Understand Capabilities – “Network”

21. Understand Capabilities – “Machines”
65 – 85 million elements
Meshing ~ 1 to 2 days
1 million ≈ 1 Gb memory
File Size → ~ 25 Gb HPC (Meshing)
Aurora (SGI Altix – 4700)
• Shared Memory - 256 GB
• 96 - 1.5 GHz Itanium 2 Processors
• SUSE Linux Enterprise Server 10
• 5.72 TB Disk Capacity
Simulation ~ 2 to 10 days
HPC (Simulation)
• Icestorm (Altix ICE 8200)
• Distributed Memory(16 GB/Node)
• 256 Nodes/8 PPN
• 2.66 GHz clock speed
• Linux OS

22. Understand Capabilities - “Software”
• SolidWorks
– Compatibility with CFD Software
– Functionality
• Design Changes
• Equations
• CD-adapco STAR-CCM+
– Physics
– Compatibility
• CAD Software
• HPC Hardware
• 3rd Party Post-Processors

23. Understand Capabilities – “Productivity”
Optimization
(i.e. Visualization & Speedup)

24. Understand Capabilities – “Productivity”
“GREY SCREEN”
Causes Results
Dedicated Master Node → I/O Loss of Data
Problems Size Problem Restart
Network/Software Limitations Visualization/Analysis Limitations

25. Understand Capabilities - Productivity
Icestorm/Altix ICE 8200 - Speedup
(19-Pin 65 Million Elements, CCM+ 3.02.003)
18.0
16.0
14.0
12.0
Speed-up
10.0
Star-CCM+
Linear
8.0
6.0
4.0
2.0
0.0
0 64 128 192 256 320 384 448 512 576 640 704 768 832 896 960 1024
Processes

26. Understand Capabilities - Architecture
e
Business
Icestorm (Altix ICE 8200)
Enclave HPC
Enclave
• Distributed Memory(16 GB/Node)
• 256 Nodes/8 PPN
• 2.66 GHz clock speed
• Linux OS

27. Understand Capabilities - Architecture
• Productivity  ~30%
– core count ↔
– double nodes
e
• Operate Machine
50% Business
Rated Capacity
Enclave HPC

– Productivity Enclave

28. Understand Capabilities – “File Size”
27 GB File Size
• Storage
– Parallel I/O
– Optimal Topology
– Save Times
• Collaboration
– File transfer
– File sharing

29. Conclusions
• M&S “Large Problems” is Challenging
– CAD/Meshing
• parallel mesher needed
– Simulation/Visualization
• Validation is Challenging
– experimental data availability
• Understand Capabilities and Limitations
– software
– HPC machines
– network infrastructure
– human infrastructure
• Key Success Factor → “SYNERGY”

30. References
1. K.D. Hamman and R.A. Berry, “A CFD M&S Process for Fast Reactor Fuel
Assemblies,” Experiments and CFD Code Applications to Nuclear Reactor
Safety (XCFD4NRS 2008), Grenoble, France, accepted, 2008.
2. CD-adapco, “Star-CCM+ (2.08.004) User Guide”, 2007.
3. N. E. Todreas and M. S. Kazimi, “Nuclear Systems I, Thermal Hydraulic
Fundamentals,” Hemisphere Publishing Corporation, 1990.
4. N. E. Todreas and M. S. Kazimi, “Nuclear Systems II, Elements of Thermal
Hydraulic Design,” Taylor and Francis, 2001.
5. Y. I. Chang, P. J. Finck, and C. Grandy, “Advanced Burner Test Reactor
Preconceptual Design Report,” ANL-ABR-1, September 2006.
6. E. H. Novendstern, Turbulent Flow Pressure Drop Model for Fuel Rod
Assemblies Utilizing a Helical Wire-Wrap Spacer System, Nuclear
Engineering Design, Vol. 22, pp. 19-27, 1972.
7. K. Rehme, Pressure Drop Correlations for Fuel Element Spacers, Nuclear
Technology, Vol. 17, pp. 15-23, 1972.
8. R. Gajapathy, et al., CFD investigation of helical wire-wrapped 7-pin fuel
bundle and the challenges in modeling full scale 217 pin bundle, Nuclear
Engineering and Design (2007), doi:10.1016/j.nucengdes.2007.05.003.
9. NPARC Alliance CFD Verification and Validation Web Site
(www.grc.nasa.gov/WWW/wind/valid/validation.html), accessed 11/1/07.