With the availability of powerful but relatively low-level plotting libraries like d3.js, plot.ly, and matplotlib, it is easier than it has ever been to create beautiful visualizations. However, these plotting libraries must be very general and thus quite complicated to accommodate arbitrarily complex plotting and visualization tasks.In this talk, I describe the plotting system used by yt, an analysis and visualization platform for volumetric data written in python. The yt plotting system wraps matplotlib, creating a domain-specific API for creating publication quality plots that matches users' intuition for how they would like to explore and visualize their data. I will provide tips for designing and testing domain-specific plotting APIs so that the resulting plots are beautiful by default, but still modifiable with the full power of the underlying plotting library.

1. NOVEMBER 15-18, 2016
#plotcon
1

2. Domain Specific Visualization
Nathan Goldbaum
Moore Data Fellow
_________
@njgoldbaum
@yt_astro
https://dxl.ncsa.illinois.edu/

3. Generic Plotting Libraries are Powerful
John Hunter Plotting Contest Entries: http://scipy2016.scipy.org/ehome/115969/276538/

4. Creative Power Implies API Complexity
4
I’m picking on matplotlib here, but the
same is true of any generic plotting library

5. 5
● Declarative Visualization
Libraries
○ Vega-lite, ggplot, altair
● Documentation by example
○ Users create plots by
cargo-culting examples
● Provide shortcuts in high-level
API
○ “Easy mode” can’t cover all
use cases
Managing API Complexity

6. 6
Alternate approach: make assumptions for users
APLpy
Astronomical Data
https://aplpy.github.io
Cartopy
Geographic Data
http://scitools.org.uk/cartopy/

7. yt: 3D Data Analysis
7
Ingest Data
Analysis/
Visualization
yt-project.org
Ramses
Gadget

8. Ingestion

9. Ingestion Representation

10. Ingestion Representation Analysis

11. Ingestion Representation Analysis Visualization

12. 12
Ingestion Representation Analysis Visualization
Science

13. 13
Ingestion Representation Analysis Visualization
Community
Science

14. Ingestion Representation Analysis Visualization
Community
Science
The ice cream sandwich of open research software

15. 15
yt: A Community of Practice
>100 Contributors as of yt 3.3.2 (Oct 2016)
Tom Abel Bili Dong Suoqing Ji Sam Leitner John Regan Geoffrey So
Gabriel Altay Nicholas Earl Allyson Julian Yuan Li Mark Richardson Casey W. Stark
Kenza Arraki Hilary Egan Christian Karch Alex Lindsay Sherwood Richers Antoine Strugarek
Kirk Barrow Daniel Fenn Max Katz Chris Malone Thomas Robitaille Elizabeth Tasker
Ricarda Beckmann John Forbes BW Keller Jonah Miller Anna Rosen Ben Thompson
Elliott Biondo Enrico Garaldi Ji-hoon Kim Joshua Moloney Chuck Rozhon Robert Thompson
Alex Bogert Sam Geen Chang-Goo Kim Christopher Moody Douglas Rudd Joseph Tomlinson
Robert Bradshaw Austin Gilbert Steffen Klemer Stuart Mumford Hsi-Yu Schive Stephanie Tonnesen
Andre Bubel Adam Ginsburg Fabian Koller Andrew Myers Anthony Scopatz Matt Turk
Corentin Cadiou Nathan Goldbaum Kacper Kowalik Jill Naiman Noel Scudder Miguel de Val-Borro
Yi-Hao Chen William Gray Mark Krumholz Desika Narayanan Patrick Shriwise Rick Wagner
Pengfei Chen Markus Haider Michael Kuhlen Kaylea Nelson Devin Silvia Mike Warren
David Collins Eric Hallman Meagan Lang Brian O'Shea Sam Skillman Charlie Watson
Brian Crosby David Hannasch Erwin Lau J.S. Oishi Stephen Skory Andrew Wetzel
Weiguang Cui Cameron Hummels Eve Lee JC Passy Aaron Smith John Wise
Andrew Cunningham Anni Järvenpää Doris Lee Rafael Ruggiero Britton Smith Michael Zingale
“Scaling a code in the human dimension”: https://arxiv.org/abs/1301.7064

16. 16
yt: A Community of Practice
Tom Abel Bili Dong Suoqing Ji Sam Leitner John Regan Geoffrey So
Gabriel Altay Nicholas Earl Allyson Julian Yuan Li Mark Richardson Casey W. Stark
Kenza Arraki Hilary Egan Christian Karch Alex Lindsay Sherwood Richers Antoine Strugarek
Kirk Barrow Daniel Fenn Max Katz Chris Malone Thomas Robitaille Elizabeth Tasker
Ricarda Beckmann John Forbes BW Keller Jonah Miller Anna Rosen Ben Thompson
Elliott Biondo Enrico Garaldi Ji-hoon Kim Joshua Moloney Chuck Rozhon Robert Thompson
Alex Bogert Sam Geen Chang-Goo Kim Christopher Moody Douglas Rudd Joseph Tomlinson
Robert Bradshaw Austin Gilbert Steffen Klemer Stuart Mumford Hsi-Yu Schive Stephanie Tonnesen
Andre Bubel Adam Ginsburg Fabian Koller Andrew Myers Anthony Scopatz Matt Turk
Corentin Cadiou Nathan Goldbaum Kacper Kowalik Jill Naiman Noel Scudder Miguel de Val-Borro
Yi-Hao Chen William Gray Mark Krumholz Desika Narayanan Patrick Shriwise Rick Wagner
Pengfei Chen Markus Haider Michael Kuhlen Kaylea Nelson Devin Silvia Mike Warren
David Collins Eric Hallman Meagan Lang Brian O'Shea Sam Skillman Charlie Watson
Brian Crosby David Hannasch Erwin Lau J.S. Oishi Stephen Skory Andrew Wetzel
Weiguang Cui Cameron Hummels Eve Lee JC Passy Aaron Smith John Wise
Andrew Cunningham Anni Järvenpää Doris Lee Rafael Ruggiero Britton Smith Michael Zingale
>100 Contributors as of yt 3.3.2 (Oct 2016)
“Scaling a code in the human dimension”: https://arxiv.org/abs/1301.7064

17. What is yt?
17

18. 18
Data on disk has no physical meaning

19. 19
With yt you can think about data using a
physically motivated interface

20. 20
Allowing you to forget about what the data look
like as a file format

21. 21
And select only the data you want to select

22. 22
And only the data you want to select

23. 23
It understands multiresolution data as a
first-class citizen

24. 24

25. 25

26. 26

27. PlotWindow plotting system
27
import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
http://yt-project.org/doc/visualizing/plots.html

28. 28

29. 29http://paste.yt-project.org/show/6913/

30. import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
Short but powerful scripts
● Register namespace
● Set up unit metadata for further
use later
30

31. import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
Short but powerful scripts
● Automagically determine data
format
● Parse parameters
● Initialize IO and data selection
interface
31

32. import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
Short but powerful scripts
● Set up in-memory mesh
● Determine available fields to plot
● Chunked IO
● Mask out overlapping data
● Convert to data CGS units
● Perform projection
● Pixelize from multi-resolution
projection
● Initialize plot object
32

33. import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(
ds, 'z', 'density', width=15*u.kpc)
prj.save()
Short but powerful scripts
● Calculate positioning and
metadata for matplotlib plot
● Create and customize matplotlib
Figure and Axes objects
● Call imshow() with appropriate
keyword arguments
● Save plot to disk
33

34. Generic plotting libraries can’t make assumptions
34
import matplotlib.pyplot as plt
plt.imshow(image_data)
plt.savefig('plot.png')

35. Customizing plots increases complexity
import matplotlib
import matplotlib.pyplot as plt
plt.imshow(image_data, norm=matplotlib.colors.LogNorm(),
origin='lower', cmap='viridis',
extent=[-7.5, 7.5, -7.5, 7.5])
cb = plt.colorbar()
cb.set_label(r'Projected Density $(rm{g}/rm{cm}^2)$')
plt.xlabel('x (kpc)')
plt.ylabel('y (kpc)')
plt.savefig('plot.png', bbox_inches='tight')
35

36. 36

37. 37
Ingestion Representation Analysis Visualization
Community
Science
Build new libraries for your use cases
Add new plotting tasks to libraries that already exist

38. Design Tips for Plotting Libraries
38

39. Provide sane defaults
39
import yt
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(ds, 'z', 'density')
prj.save('./frames/0000.png')
for i in range(150):
prj.zoom(1.03)
prj.save('./frames/%04d.png' % (i+1))

40. import yt
ds = yt.load('DD0600/DD0600')
prj = yt.ProjectionPlot(ds, 'z', 'density')
prj.set_zlim('density', 8e-6, 3e-1)
prj.save('./frames/0000.png')
for i in range(150):
prj.zoom(1.03)
prj.save('./frames/%04d.png' % (i+1))
Customize via properties/setters, not keyword arguments
40

41. Don’t overcomplicate for niche use cases
import yt
import yt.units as u
ds = yt.load('DD0600/DD0600')
fields = ['density', 'temperature',
'Velocity_divergence', 'grid_level']
slc = yt.SlicePlot(ds, 'z', fields)
slc.set_width(15*kpc)
slc.set_log('grid_level', False)
slc.set_unit('velocity_divergence', 'km/s/kpc')
slc.set_unit('density', 'Msun/pc**3')
slc.save()
4
1

42. Expose base plotting library primitives
slc = yt.SlicePlot(ds, 'z', 'density')
plot = slc.plots['density']
# matplotlib.figure.Figure
plot.figure
# matplotlib.axes.Axes
plot.axes
# matlotlib.colorbar.Colorbar
plot.cb
# matplotlib.image.AxesImage
plot.image
42http://yt-project.org/docs/dev/cookbook/simple_plots.html#matplotlib-primitives

43. Only perform expensive actions as needed
43
def invalidate_plot(f):
@functools.wraps(f)
def newfunc(*args, **kwargs):
rv = f(*args, **kwargs)
args[0]._plot_valid = False
return rv
return newfunc
def validate_plot(f):
@wraps(f)
def newfunc(*args, **kwargs):
if not args[0]._plot_valid:
args[0]._setup_plots()
rv = f(*args, **kwargs)
return rv
return newfunc
class SlicePlot:
def _setup_plots(self):
# make plots using matplotlib primitives
@invalidate_plot
def set_axes_unit(self, unit_name):
self._axes_unit_names = (unit_name, unit_name)
@validate_plot
def save(self):
for field in self.plots:
self.plots[field].save()

44. 44

45. Thank you for your attention!
45
Questions? Concerns? Comments?