This talk is a "how I did it" talk about how I took an idea, a web cam, Python, Django, and the Python Imaging Library and created art, explored science, and illustrated concepts that our ancestors knew by watching the sky but we have lost.

1. Observational Science
With Python
And a Webcam
By: Eric Floehr
Observational Science With Python and a Webcam by Eric Floehr is licensed under a
Creative Commons Attribution-ShareAlike 3.0 Unported License.

2. So I had a webcam...

3. and I made a time-lapse video.
http://bit.ly/ospw-1

4. I wanted to do better.
Longer
Inside location
Automated
Once a minute

5. So this is what I did.
Second floor window
Little-used room
Pointing west
Pull pictures down with cronjob
That runs once per minute

6. And it looks like this.

7. Two years later...

8. I have...
● 896,309 image files
● 11,809,972,880 bytes
● 10.9989 gigabytes
● From 2:14pm on August 29, 2010
● To 4:11pm on July 25, 2012
● Still going...

9. Tenet #1:
Don't be afraid of big data.

10. What can we do?
● Moar Time-lapse!
● Explore phenomenon that occurs
over long periods
● Unique visualizations
● Have lots of fun!

11. First...
We need to organize the data.

12. Database!

13. How will we access the data?
Let's use Django!

14. Why Django?
● It has a good ORM
● It has a command framework
● Makes extending to the web later
easy
● I already am familiar with it
● Django isn't just for web :-)

15. Quick setup
● Create your virtualenv (with site packages)
● PIL is a pain to compile
● Install Django
● django-admin.py startproject
● Edit settings.py
● createdb pics
● django-admin.py startapp pics
● django-admin.py syncdb

16. Tenet #2:
Don't let small things keep you
from your big picture.

17. What do we need to store?
● Image file location and filename
● Time the image was taken
● Interesting information about the
image
● Is it a valid image?

18. class Picture(models.Model):
filepath = models.CharField(max_length=1024)
filename = models.CharField(max_length=1024)
timestamp = models.DateTimeField(db_index=True)
file_timestamp = models.DateTimeField(null=True)
# Auto-populated, don't manually enter
hour = models.SmallIntegerField(null=True, db_index=True)
minute = models.SmallIntegerField(null=True, db_index=True)
file_hour = models.SmallIntegerField(null=True)
file_minute = models.SmallIntegerField(null=True)
# End auto-populated fields
size = models.IntegerField(default=0)
center_color = models.IntegerField(null=True)
mean_color = models.IntegerField(null=True)
median_color = models.IntegerField(null=True)
stddev_red = models.IntegerField(null=True)
stddev_green = models.IntegerField(null=True)
stddev_blue = models.IntegerField(null=True)
min_color = models.IntegerField(null=True)
max_color = models.IntegerField(null=True)
valid = models.BooleanField(default=False)
class Meta:
ordering = ['timestamp']

19. class Picture(models.Model):
filepath = models.CharField(max_length=1024)
filename = models.CharField(max_length=1024) <-- File location and name
timestamp = models.DateTimeField(db_index=True)
file_timestamp = models.DateTimeField(null=True)
# Auto-populated, don't manually enter
hour = models.SmallIntegerField(null=True, db_index=True)
minute = models.SmallIntegerField(null=True, db_index=True)
file_hour = models.SmallIntegerField(null=True)
file_minute = models.SmallIntegerField(null=True)
# End auto-populated fields
size = models.IntegerField(default=0)
center_color = models.IntegerField(null=True)
mean_color = models.IntegerField(null=True)
median_color = models.IntegerField(null=True)
stddev_red = models.IntegerField(null=True)
stddev_green = models.IntegerField(null=True)
stddev_blue = models.IntegerField(null=True)
min_color = models.IntegerField(null=True)
max_color = models.IntegerField(null=True)
valid = models.BooleanField(default=False)
class Meta:
ordering = ['timestamp']

20. class Picture(models.Model):
filepath = models.CharField(max_length=1024)
filename = models.CharField(max_length=1024)
timestamp = models.DateTimeField(db_index=True)
file_timestamp = models.DateTimeField(null=True)
# Auto-populated, don't manually enter
hour = models.SmallIntegerField(null=True, db_index=True) <-- Image Timestamp
minute = models.SmallIntegerField(null=True, db_index=True)
file_hour = models.SmallIntegerField(null=True)
file_minute = models.SmallIntegerField(null=True)
# End auto-populated fields
size = models.IntegerField(default=0)
center_color = models.IntegerField(null=True)
mean_color = models.IntegerField(null=True)
median_color = models.IntegerField(null=True)
stddev_red = models.IntegerField(null=True)
stddev_green = models.IntegerField(null=True)
stddev_blue = models.IntegerField(null=True)
min_color = models.IntegerField(null=True)
max_color = models.IntegerField(null=True)
valid = models.BooleanField(default=False)
class Meta:
ordering = ['timestamp']

21. class Picture(models.Model):
filepath = models.CharField(max_length=1024)
filename = models.CharField(max_length=1024)
timestamp = models.DateTimeField(db_index=True)
file_timestamp = models.DateTimeField(null=True)
# Auto-populated, don't manually enter
hour = models.SmallIntegerField(null=True, db_index=True)
minute = models.SmallIntegerField(null=True, db_index=True)
file_hour = models.SmallIntegerField(null=True)
file_minute = models.SmallIntegerField(null=True)
# End auto-populated fields
size = models.IntegerField(default=0)
center_color = models.IntegerField(null=True)
mean_color = models.IntegerField(null=True)
median_color = models.IntegerField(null=True)
stddev_red = models.IntegerField(null=True) <-- Interesting image data
stddev_green = models.IntegerField(null=True)
stddev_blue = models.IntegerField(null=True)
min_color = models.IntegerField(null=True)
max_color = models.IntegerField(null=True)
valid = models.BooleanField(default=False)
class Meta:
ordering = ['timestamp']

22. class Picture(models.Model):
filepath = models.CharField(max_length=1024)
filename = models.CharField(max_length=1024)
timestamp = models.DateTimeField(db_index=True)
file_timestamp = models.DateTimeField(null=True)
# Auto-populated, don't manually enter
hour = models.SmallIntegerField(null=True, db_index=True)
minute = models.SmallIntegerField(null=True, db_index=True)
file_hour = models.SmallIntegerField(null=True)
file_minute = models.SmallIntegerField(null=True)
# End auto-populated fields
size = models.IntegerField(default=0)
center_color = models.IntegerField(null=True)
mean_color = models.IntegerField(null=True)
median_color = models.IntegerField(null=True)
stddev_red = models.IntegerField(null=True)
stddev_green = models.IntegerField(null=True)
stddev_blue = models.IntegerField(null=True)
min_color = models.IntegerField(null=True)
max_color = models.IntegerField(null=True)
valid = models.BooleanField(default=False)
<-- Is it valid? How do we order?
class Meta:
ordering = ['timestamp']

23. Loading the Data

24. How do we populate the table?
● Iterate through directories of
image files
● Parse the file name to get
timestamp
● Get file timestamp to compare
● Load image to collect information
about the image

25. Find the image files
import os
import fnmatch
import datetime
for dirpath, dirs, files in os.walk(directory):
for f in sorted(fnmatch.filter(files, 'image-*.jpg')):
# Pull out timestamp
timestamp = f.split('.')[0].split('-')[1]
date = datetime.datetime.fromtimestamp(int(timestamp), utc)

26. Find the image files
import os
import fnmatch
import datetime
for dirpath, dirs, files in os.walk(directory):
for f in sorted(fnmatch.filter(files, 'image-*.jpg')):
# Pull out timestamp
timestamp = f.split('.')[0].split('-')[1]
date = datetime.datetime.fromtimestamp(int(timestamp), utc)

27. Find the image files
import os
import fnmatch
import datetime
for dirpath, dirs, files in os.walk(directory):
for f in sorted(fnmatch.filter(files, 'image-*.jpg')):
# Pull out timestamp
timestamp = f.split('.')[0].split('-')[1]
date = datetime.datetime.fromtimestamp(int(timestamp), utc)

28. Find the image files
import os
import fnmatch
import datetime
for dirpath, dirs, files in os.walk(directory):
for f in sorted(fnmatch.filter(files, 'image-*.jpg')):
# Pull out timestamp
timestamp = f.split('.')[0].split('-')[1]
date = datetime.datetime.fromtimestamp(int(timestamp), utc)

29. Find the image files
import os
import fnmatch
import datetime
for dirpath, dirs, files in os.walk(directory):
for f in sorted(fnmatch.filter(files, 'image-*.jpg')):
# Pull out timestamp
timestamp = f.split('.')[0].split('-')[1]
date = datetime.datetime.fromtimestamp(int(timestamp), utc)

30. Use PIL to get image info
import Image, ImageStat
From util.color import rgb_to_int
try:
im = Image.open(pic.filepath)
stats = ImageStat.Stat(im)
pic.center_color = rgb_to_int(im.getpixel((320,240)))
pic.mean_color = rgb_to_int(stats.mean)
pic.median_color = rgb_to_int(stats.median)
if im.size <> (640,480):
pic.valid = False
except:
pic.valid = False

31. Use PIL to get image info
import Image, ImageStat
From util.color import rgb_to_int
try:
im = Image.open(pic.filepath)
stats = ImageStat.Stat(im)
pic.center_color = rgb_to_int(im.getpixel((320,240)))
pic.mean_color = rgb_to_int(stats.mean)
pic.median_color = rgb_to_int(stats.median)
if im.size <> (640,480):
pic.valid = False
except:
pic.valid = False

32. Use PIL to get image info
import Image, ImageStat
From util.color import rgb_to_int
try:
im = Image.open(pic.filepath)
stats = ImageStat.Stat(im)
pic.center_color = rgb_to_int(im.getpixel((320,240)))
pic.mean_color = rgb_to_int(stats.mean)
pic.median_color = rgb_to_int(stats.median)
if im.size <> (640,480):
pic.valid = False
except:
pic.valid = False

33. Use PIL to get image info
import Image, ImageStat
From util.color import rgb_to_int
try:
im = Image.open(pic.filepath)
stats = ImageStat.Stat(im)
pic.center_color = rgb_to_int(im.getpixel((320,240)))
pic.mean_color = rgb_to_int(stats.mean)
pic.median_color = rgb_to_int(stats.median)
if im.size <> (640,480):
pic.valid = False
except:
pic.valid = False

34. It took a an hour or two using
six threads on my desktop

35. Tenet #3:
You have a 1990's
supercomputer on your lap or
under your desk.
Use it!

36. Let's Explore the Data

37. Size Indicates Complexity in jpeg
pics=# select timestamp, filepath, size from pics_picture where size=(select
max(size) from pics_picture);
timestamp | filepath | size
------------------------+--------------------------------------+-------
2011-10-10 18:26:01-04 | /data/pics/1318/image-1318285561.jpg | 64016
http://bit.ly/ospw-2

38. High Stddev Means Color Variation
pics=# select timestamp, filepath, size from pics_picture where
stddev_red+stddev_green+stddev_blue =
(select max(stddev_red+stddev_green+stddev_blue) from pics_picture);
Timestamp | filepath | size
------------------------+--------------------------------------+-------
2011-09-29 17:20:01-04 | /data/pics/1317/image-1317331201.jpg | 22512
http://bit.ly/ospw-3

39. About the Dataset
pics=# select min(timestamp) as start, max(timestamp) as end from pics_picture;
start | end
------------------------+------------------------
2010-08-29 14:14:01-04 | 2012-07-25 16:11:01-04
(1 row)
pics=# select count(*), sum(case when valid=false then 1 else 0 end) as invalid
from pics_picture;
count | invalid
--------+---------
896309 | 29555
(1 row)
pics=# select timestamp, filepath, size from pics_picture where size>0 and
valid=false order by size desc limit 1;
timestamp | filepath | size
------------------------+--------------------------------------+-------
2012-04-25 08:16:01-04 | /data/pics/1335/image-1335356161.jpg | 39287
(1 row)
http://bit.ly/ospw-4

40. Yuck! This Data Sucks!
● 29,555 invalid image files
● That's 3.3% of all image files
● Worse yet, there isn't a file every minute
● Based on start and end times, we should
have 1,002,358 images
● We are missing 10.6% of all minutes
between start and end times

41. It's Worse...I Forgot The Bolts!
http://bit.ly/ospw-5
http://bit.ly/ospw-6
http://bit.ly/ospw-7

42. * Interestingly, I was listening to the Serious Business remix of “All Is Not
Lost” by OK Go from the Humble Music Bundle as I made the previous slide.

43. Tenet #4:
Real world data is messy.
That's ok. Just work around it.

44. How we can work around it?
● Create table of all minutes
● Accept images “near” missing
minutes
● Use empty images when
acceptable images can't be found

45. Let's Make Time-lapse Movies

46. How do we make movies?
● Collect a set of images in frame
order.
● Send that list of images to a movie
maker.
● Wait for movie to be made.
● Watch movie!

47. The previous bullet points in code
from movies import ImageSequence
from pics.models import NormalizedPicture
import datetime
from pytz import timezone
ims = ImageSequence()
ettz = timezone('US/Eastern')
start = datetime.datetime(2011,4,7).replace(tzinfo=ettz)
end = start + datetime.timedelta(days=1)
pics = NormalizedPicture.objects.filter(timestamp__gte=start,
timestamp__lt=end)
ims = ImageSequence()
for pic in pics:
if pic.picture is not None:
ims.add_picture(pic.picture)
else:
ims.add_image('/tmp/no_image.png')
ims.make_timelapse('/tmp/{0}'.format(start.strftime("%Y%m%d")), fps=24)

48. Collect a set of Images
from movies import ImageSequence
from pics.models import NormalizedPicture
import datetime
from pytz import timezone
ims = ImageSequence()
ettz = timezone('US/Eastern')
start = datetime.datetime(2011,4,7).replace(tzinfo=ettz)
end = start + datetime.timedelta(days=1)
pics = NormalizedPicture.objects.filter(timestamp__gte=start,
timestamp__lt=end)
ims = ImageSequence()
for pic in pics:
if pic.picture is not None:
ims.add_picture(pic.picture)
else:
ims.add_image('/tmp/no_image.png')
ims.make_timelapse('/tmp/{0}'.format(start.strftime("%Y%m%d")), fps=24)

49. Send Images to Movie Maker
from movies import ImageSequence
from pics.models import NormalizedPicture
import datetime
from pytz import timezone
ims = ImageSequence()
ettz = timezone('US/Eastern')
start = datetime.datetime(2011,4,7).replace(tzinfo=ettz)
end = start + datetime.timedelta(days=1)
pics = NormalizedPicture.objects.filter(timestamp__gte=start,
timestamp__lt=end)
ims = ImageSequence()
for pic in pics:
if pic.picture is not None:
ims.add_picture(pic.picture)
else:
ims.add_image('/tmp/no_image.png')
ims.make_timelapse('/tmp/{0}'.format(start.strftime("%Y%m%d")), fps=24)

50. Wait For Movie To Be Made
from movies import ImageSequence
from pics.models import NormalizedPicture
import datetime
from pytz import timezone
ims = ImageSequence()
ettz = timezone('US/Eastern')
start = datetime.datetime(2011,4,7).replace(tzinfo=ettz)
end = start + datetime.timedelta(days=1)
pics = NormalizedPicture.objects.filter(timestamp__gte=start,
timestamp__lt=end)
ims = ImageSequence()
for pic in pics:
if pic.picture is not None:
ims.add_picture(pic.picture)
else:
ims.add_image('/tmp/no_image.png')
ims.make_timelapse('/tmp/{0}'.format(start.strftime("%Y%m%d")), fps=24)

51. Wait For Movie To Be Made
class ImageSequence(object):
def __init__(self):
self.images = []
def add_picture(self, picture):
self.images.append(picture.filepath)
def write_to_file(self, fileobj):
for image in self.images:
fileobj.write(image)
def make_timelapse(self, name, fps=25):
tmpfile = tempfile.NamedTemporaryFile()
self.write_to_file(tmpfile)
bitrate = int(round(60 * fps * 640 * 480 / 256.0))
execall = []
execall.append(mencoder_exe)
execall.extend(mencoder_options)
execall.extend(["-lavcopts",
"vcodec=mpeg4:vbitrate={0}:mbd=2:keyint=132:v4mv:vqmin=3:lumi_mask=0.07:dark_mask=0.2
:mpeg_quant:scplx_mask=0.1:tcplx_mask=0.1:naq".format(bitrate)])
execall.append("mf://@{0}".format(tmpfile.name))
execall.extend(["-mf", "type=jpeg:fps={0}".format(fps)])
execall.extend(["-o", "{name}.mp4".format(name=name)])
return subprocess.call(execall)

52. Watch Movie!
http://bit.ly/ospw-8
http://bit.ly/ospw-8-raw

53. We aren't limited to
consecutive minutes...

54. Movie of a Specific Time
from movies import ImageSequence
from pics.models import NormalizedPicture
ims = ImageSequence()
Hour = 22 # UTC Time
minute = 0
last_pic = None
pics = NormalizedPicture.objects.filter(hour=hour, minute=minute)
last_pic = None
for pic in pics:
if pic.picture is not None:
ims.add_picture(pic.picture)
last_pic = pic.picture
else:
# Use yesterdays' image
if last_pic is not None:
ims.add_picture(last_pic)
else: # Give up
ims.add_image('/tmp/no_image.jpg')
ims.make_timelapse('/tmp/time_{0:02d}{1:02d}'.format(hour,minute),fps=12)

55. Movie of a Specific Time
from movies import ImageSequence
from pics.models import NormalizedPicture
ims = ImageSequence()
Hour = 22 # UTC Time
minute = 0
last_pic = None
pics = NormalizedPicture.objects.filter(hour=hour, minute=minute)
last_pic = None
for pic in pics:
if pic.picture is not None:
ims.add_picture(pic.picture)
last_pic = pic.picture
else:
# Use yesterdays' image
if last_pic is not None:
ims.add_picture(last_pic)
else: # Give up
ims.add_image('/tmp/no_image.jpg')
ims.make_timelapse('/tmp/time_{0:02d}{1:02d}'.format(hour,minute),fps=12)

56. Movie of a Specific Time
from movies import ImageSequence
from pics.models import NormalizedPicture
ims = ImageSequence()
hour = 22 # UTC time
minute = 0
last_pic = None
pics = NormalizedPicture.objects.filter(hour=hour, minute=minute)
last_pic = None
for pic in pics:
if pic.picture is not None:
ims.add_picture(pic.picture)
last_pic = pic.picture
else:
# Use yesterdays' image
if last_pic is not None:
ims.add_picture(last_pic)
else: # Give up
ims.add_image('/tmp/no_image.jpg')
ims.make_timelapse('/tmp/time_{0:02d}{1:02d}'.format(hour,minute),fps=12)

57. Watch Movie!
http://bit.ly/ospw-9
http://bit.ly/ospw-9-raw

58. Now what if we want the Sun
the same height above the
horizon, so we can better see
the seasonal progression of
the Sun?

59. We can anchor on sunset. At a
given time before sunset, the
Sun will be at relatively the
same height above the
horizon.

60. Where I thank Brandon Craig
Rhodes for pyephem

61. Movie at Specific Time Before Sunset
import sky
obs = sky.webcam()
while current < end:
# Get today's sunset time, but 70 minutes before
pic_time = sky.sunset(obs, current) - datetime.timedelta(minutes=70)
pic_time = pic_time.replace(second=0, microsecond=0)
pic = NormalizedPicture.objects.get(timestamp=pic_time)
if pic.picture is not None:
ims.add_picture(pic.picture)
else:
piclist = NormalizedPicture.objects.get(
timestamp__gte=pic_time - datetime.timedelta(minutes=20),
timestamp__lte=pic_time + datetime.timedelta(minutes=20),
picture__id__isnull=False)
if len(piclist) > 0:
ims.add_picture(piclist[0].picture)
else:
# Use yesterdays' image
if last_pic is not None:
ims.add_picture(last_pic)
else: # Give up
ims.add_image('/tmp/no_image.jpg')
current = current + datetime.timedelta(days=1)

62. Movie at Specific Time Before Sunset
import sky
obs = sky.webcam()
while current < end:
# Get today's sunset time, but 70 minutes before
pic_time = sky.sunset(obs, current) - datetime.timedelta(minutes=70)
pic_time = pic_time.replace(second=0, microsecond=0)
pic = NormalizedPicture.objects.get(timestamp=pic_time)
if pic.picture is not None:
ims.add_picture(pic.picture)
else:
piclist = NormalizedPicture.objects.get(
timestamp__gte=pic_time - datetime.timedelta(minutes=20),
timestamp__lte=pic_time + datetime.timedelta(minutes=20),
picture__id__isnull=False)
if len(piclist) > 0:
ims.add_picture(piclist[0].picture)
else:
# Use yesterdays' image
if last_pic is not None:
ims.add_picture(last_pic)
else: # Give up
ims.add_image('/tmp/no_image.jpg')
current = current + datetime.timedelta(days=1)

63. Our sky module that uses pyephem
import ephem
from pytz import timezone, utc
sun = ephem.Sun()
moon = ephem.Moon()
est = timezone('EST')
def observer():
location = ephem.Observer()
location.lon = '-83:23:24'
location.lat = '40:13:48'
location.elevation = 302
return location
def sunrise(observer, date):
observer.date = date.astimezone(utc)
return observer.next_rising(sun).datetime().replace(tzinfo=utc).astimezone(est)
def sunset(observer, date):
observer.date = date.astimezone(utc)
return observer.next_setting(sun).datetime().replace(tzinfo=utc).astimezone(est)
def moonset(observer, date):
observer.date = date.astimezone(utc)
return observer.next_setting(moon).datetime().replace(tzinfo=utc).astimezone(est)

64. Our sky module that uses pyephem
import ephem
from pytz import timezone, utc
sun = ephem.Sun()
moon = ephem.Moon()
est = timezone('EST')
def observer():
location = ephem.Observer()
location.lon = '-83:23:24'
location.lat = '40:13:48'
location.elevation = 302
return location
def sunrise(observer, date):
observer.date = date.astimezone(utc)
return observer.next_rising(sun).datetime().replace(tzinfo=utc).astimezone(est)
def sunset(observer, date):
observer.date = date.astimezone(utc)
return observer.next_setting(sun).datetime().replace(tzinfo=utc).astimezone(est)
def moonset(observer, date):
observer.date = date.astimezone(utc)
return observer.next_setting(moon).datetime().replace(tzinfo=utc).astimezone(est)

65. Our sky module that uses pyephem
import ephem
from pytz import timezone, utc
sun = ephem.Sun()
moon = ephem.Moon()
est = timezone('EST')
def observer():
location = ephem.Observer()
location.lon = '-83:23:24'
location.lat = '40:13:48'
location.elevation = 302
return location
def sunrise(observer, date):
observer.date = date.astimezone(utc)
return observer.next_rising(sun).datetime().replace(tzinfo=utc).astimezone(est)
def sunset(observer, date):
observer.date = date.astimezone(utc)
return observer.next_setting(sun).datetime().replace(tzinfo=utc).astimezone(est)
def moonset(observer, date):
observer.date = date.astimezone(utc)
return observer.next_setting(moon).datetime().replace(tzinfo=utc).astimezone(est)

66. Watch Movie!
http://bit.ly/ospw-10
http://bit.ly/ospw-10-raw

67. August 29, 2010
6:59pm EDT
http://bit.ly/ospw-11
October 22, 2010
5:33pm EDT
http://bit.ly/ospw-12

68. Ancient Astronomical Alignments
http://bit.ly/ospw-14
Photo credits:
http://en.wikipedia.org/wiki/File:ChichenItzaEquinox.jpg
http://www.colorado.edu/Conferences/chaco/tour/images/dagger.jpg

69. Tenet #5:
Once you have a foundation
(data or code), however messy,
you can build higher.

70. So let's build higher!

71. We Also Take Pictures At Night
● Could I have captured a UFO in an
image?
● Or a fireball?
● Some other phenomenon?
● What track does the moon take
through the sky?
● How about Venus or Jupiter?

72. Moon Tracks and UFOs
● We want to find interesting things
● How do we easily identify “interesting things”
● At night, bright things are interesting things
● Brightness is a good proxy for “interesting”
● It makes it easy to identify interesting things
● As it is easier to spot black spots on white images,
we'll invert the images

73. Making Night Tracks
● Process each image
● Stack images into a single “all
night” image
● From one hour after sunset to one
hour before sunrise the next day
● Black splotches are interesting
things

74. def make_nighttrack(rootdir, day):
obs = sky.observer()
# One hour after sunset
start_time = sky.sunset(obs, day) + datetime.timedelta(hours=1)
# Until one hour before sunrise the next day
end_time = sky.sunrise(obs, tomorrow) - datetime.timedelta(hours=1)
pics = NormalizedPicture.objects.filter(
timestamp__gte=start_time, timestamp__lt=end_time, picture__id__isnull=False)
print "Drawing image with {0} images".format(len(pics))
im = Image.new("L", (640,480), background_color)
for picture in pics:
source = Image.open(picture.picture.filepath)
# Get the negative
source_gray = ImageOps.grayscale(source)
source_neg = ImageOps.invert(source_gray)
# Threshold white
source_thresh = Image.eval(source_neg, lambda x: 255*(x>224))
# Merge in the new image
im = ImageChops.multiply(im, source_thresh)
# Put a date on the image
canvas = ImageDraw.Draw(im)
canvas.text((10,460), day.strftime("%Y-%m-%d"))
# And save
im.save('{root}/{date}_mt.png'.format(root=rootdir,date=day.strftime("%Y%m%d")))

75. def make_nighttrack(rootdir, day):
obs = sky.observer()
# One hour after sunset
start_time = sky.sunset(obs, day) + datetime.timedelta(hours=1)
# Until one hour before sunrise the next day
end_time = sky.sunrise(obs, tomorrow) - datetime.timedelta(hours=1)
pics = NormalizedPicture.objects.filter(
timestamp__gte=start_time, timestamp__lt=end_time, picture__id__isnull=False)
print "Drawing image with {0} images".format(len(pics))
im = Image.new("L", (640,480), background_color)
for picture in pics:
source = Image.open(picture.picture.filepath)
# Get the negative
source_gray = ImageOps.grayscale(source)
source_neg = ImageOps.invert(source_gray)
# Threshold white
source_thresh = Image.eval(source_neg, lambda x: 255*(x>224))
# Merge in the new image
im = ImageChops.multiply(im, source_thresh)
# Put a date on the image
canvas = ImageDraw.Draw(im)
canvas.text((10,460), day.strftime("%Y-%m-%d"))
# And save
im.save('{root}/{date}_mt.png'.format(root=rootdir,date=day.strftime("%Y%m%d")))

76. def make_nighttrack(rootdir, day):
obs = sky.observer()
# One hour after sunset
start_time = sky.sunset(obs, day) + datetime.timedelta(hours=1)
# Until one hour before sunrise the next day
end_time = sky.sunrise(obs, tomorrow) - datetime.timedelta(hours=1)
pics = NormalizedPicture.objects.filter(
timestamp__gte=start_time, timestamp__lt=end_time, picture__id__isnull=False)
print "Drawing image with {0} images".format(len(pics))
im = Image.new("L", (640,480), background_color)
for picture in pics:
source = Image.open(picture.picture.filepath)
# Get the negative
source_gray = ImageOps.grayscale(source)
source_neg = ImageOps.invert(source_gray)
# Threshold white
source_thresh = Image.eval(source_neg, lambda x: 255*(x>224))
# Merge in the new image
im = ImageChops.multiply(im, source_thresh)
# Put a date on the image
canvas = ImageDraw.Draw(im)
canvas.text((10,460), day.strftime("%Y-%m-%d"))
# And save
im.save('{root}/{date}_mt.png'.format(root=rootdir,date=day.strftime("%Y%m%d")))

77. def make_nighttrack(rootdir, day):
obs = sky.observer()
# One hour after sunset
start_time = sky.sunset(obs, day) + datetime.timedelta(hours=1)
# Until one hour before sunrise the next day
end_time = sky.sunrise(obs, tomorrow) - datetime.timedelta(hours=1)
pics = NormalizedPicture.objects.filter(
timestamp__gte=start_time, timestamp__lt=end_time, picture__id__isnull=False)
print "Drawing image with {0} images".format(len(pics))
im = Image.new("L", (640,480), background_color)
for picture in pics:
source = Image.open(picture.picture.filepath)
# Get the negative
source_gray = ImageOps.grayscale(source)
source_neg = ImageOps.invert(source_gray)
# Threshold white
source_thresh = Image.eval(source_neg, lambda x: 255*(x>224))
# Merge in the new image
im = ImageChops.multiply(im, source_thresh)
# Put a date on the image
canvas = ImageDraw.Draw(im)
canvas.text((10,460), day.strftime("%Y-%m-%d"))
# And save
im.save('{root}/{date}_mt.png'.format(root=rootdir,date=day.strftime("%Y%m%d")))

78. Venus, Jupiter, and the Moon
http://bit.ly/ospw-15

79. Lighting Up the Sky
http://bit.ly/ospw-16

80. Lighting Up the Sky
http://bit.ly/ospw-17

81. Lightning Bug? Aircraft? UFO!
Venus
?
Moon
http://bit.ly/ospw-18
http://bit.ly/ospw-19
http://bit.ly/ospw-20

82. Shows up 3 weeks later
http://bit.ly/ospw-21
http://bit.ly/ospw-22
http://bit.ly/ospw-23

83. Last Oddity: 9/2/2011
http://bit.ly/ospw-34

84. Last Oddity: 9/2/2011
Single-Frame Oddity Crescent Moon That Night
http://bit.ly/ospw-30 http://bit.ly/ospw-32
http://bit.ly/ospw-31 http://bit.ly/ospw-33

85. Let's make some science art!

86. Daystrips
● So far we've been using whole
images...let's change that
● Instead of using a whole image,
let's just use one line
● Kind of like a scanner
● Start at midnight, stacking lines

87. Daystrip Scanner
12:06pm
11:40pm http://bit.ly/ospw-24

88. Daystrips
● There are 1440 minutes in a day
● Images will be 1440 pixels high
● We place image line at the current
minute in the day
● HOUR * 60 + MINUTE

89. This is beginning to look familiar
def make_daystrip(rootdir, day):
end = day + datetime.timedelta(days=1)
print "Getting data for {0}".format(day)
pics = NormalizedPicture.objects.filter(
timestamp__gte=day, timestamp__lt=end, picture__id__isnull=False)
im = Image.new('RGB', (640,1440), background_color)
for picture in pics:
source = Image.open(picture.picture.filepath)
# Get row 1/3 of the way down, painting on proper row of canvas
timestamp = picture.timestamp.astimezone(esttz)
y = timestamp.hour * 60 + timestamp.minute
im.paste(source.crop((0,160,640,161)),(0,y))
im.save('{root}/{date}_daystrip.png'.format(root=rootdir,date=day.strftime("%Y%m
%d")))

90. Create Our New Image
def make_daystrip(rootdir, day):
end = day + datetime.timedelta(days=1)
print "Getting data for {0}".format(day)
pics = NormalizedPicture.objects.filter(
timestamp__gte=day, timestamp__lt=end, picture__id__isnull=False)
im = Image.new('RGB', (640,1440), background_color)
for picture in pics:
source = Image.open(picture.picture.filepath)
# Get row 1/3 of the way down, painting on proper row of canvas
timestamp = picture.timestamp.astimezone(esttz)
y = timestamp.hour * 60 + timestamp.minute
im.paste(source.crop((0,160,640,161)),(0,y))
im.save('{root}/{date}_daystrip.png'.format(root=rootdir,date=day.strftime("%Y%m
%d")))

91. Iterate Though Our Images
def make_daystrip(rootdir, day):
end = day + datetime.timedelta(days=1)
print "Getting data for {0}".format(day)
pics = NormalizedPicture.objects.filter(
timestamp__gte=day, timestamp__lt=end, picture__id__isnull=False)
im = Image.new('RGB', (640,1440), background_color)
for picture in pics:
source = Image.open(picture.picture.filepath)
# Get row 1/3 of the way down, painting on proper row of canvas
timestamp = picture.timestamp.astimezone(esttz)
y = timestamp.hour * 60 + timestamp.minute
im.paste(source.crop((0,160,640,161)),(0,y))
im.save('{root}/{date}_daystrip.png'.format(root=rootdir,date=day.strftime("%Y%m
%d")))

92. And Paste The Single Row
def make_daystrip(rootdir, day):
end = day + datetime.timedelta(days=1)
print "Getting data for {0}".format(day)
pics = NormalizedPicture.objects.filter(
timestamp__gte=day, timestamp__lt=end, picture__id__isnull=False)
im = Image.new('RGB', (640,1440), background_color)
for picture in pics:
source = Image.open(picture.picture.filepath)
# Get row 1/3 of the way down, painting on proper row of canvas
timestamp = picture.timestamp.astimezone(esttz)
y = timestamp.hour * 60 + timestamp.minute
im.paste(source.crop((0,160,640,161)),(0,y))
im.save('{root}/{date}_daystrip.png'.format(root=rootdir,date=day.strftime("%Y%m
%d")))

93. Finally, save the image
def make_daystrip(rootdir, day):
end = day + datetime.timedelta(days=1)
print "Getting data for {0}".format(day)
pics = NormalizedPicture.objects.filter(
timestamp__gte=day, timestamp__lt=end, picture__id__isnull=False)
im = Image.new('RGB', (640,1440), background_color)
for picture in pics:
source = Image.open(picture.picture.filepath)
# Get row 1/3 of the way down, painting on proper row of canvas
timestamp = picture.timestamp.astimezone(esttz)
y = timestamp.hour * 60 + timestamp.minute
im.paste(source.crop((0,160,640,161)),(0,y))
im.save('{root}/{date}_daystrip.png'.format(root=rootdir,date=day.strftime("%Y%m
%d")))

94. Daystrip Example – March 17, 2011
Midnight
Moon Crossing
Sunrise
More cloudy (gray)
Less cloudy (blue)
Sun Crossing
Sunset
Midnight
http://bit.ly/ospw-25

95. Shortest and Longest Day
Dec 22, 2011 June 20, 2012
http://bit.ly/ospw-26 http://bit.ly/ospw-27

96. Tenet #6:
Don't be afraid to be creative.
Science can be beautiful.

97. Everything we've made so far
spans a day or less.

98. What we've done so far
● Viewed full images of interest
● Combined full images in movies
● Stacked full images to find UFOs
● Took a full line from a day's worth
of images
● Everything is a day or less of data

99. Let's use ALL the days!

100. What if...

101. Instead of an image row being
a single minute...
it was a whole day...

102. And each pixel in the row was
a minute in the day.

103. Therefore, each of our 896,309
webcam images would
comprise a single pixel in our
über-image.

104. A Visual Representation
Midnight
Midnight
Noon
Minutes in a day (1440)
Start Day
Days (each row is one day)
Each pixel is one
minute in the day
Image
End Day

105. pics = NormalizedPicture.objects.all()
im = Image.new('RGB', (1440,num_days), background_color)
Canvas = ImageDraw.Draw(im)
for picture in pics:
# Get XY
timestamp = picture.timestamp.astimezone(est)
y_timedelta = timestamp - start
y = y_timedelta.days
x = timestamp.hour * 60 + timestamp.minute
# Paint pixel
if picture.picture is not None:
color = int_to_rgb(picture.picture.median_color)
else:
color = background_color
canvas.point((x,y), fill=color)
# All done, save
im.save(filepath)

106. pics = NormalizedPicture.objects.all()
im = Image.new('RGB', (1440,num_days), background_color)
Canvas = ImageDraw.Draw(im)
for picture in pics:
# Get XY
timestamp = picture.timestamp.astimezone(est)
y_timedelta = timestamp - start
y = y_timedelta.days
x = timestamp.hour * 60 + timestamp.minute
# Paint pixel
if picture.picture is not None:
color = int_to_rgb(picture.picture.median_color)
else:
color = background_color
canvas.point((x,y), fill=color)
# All done, save
im.save(filepath)

107. pics = NormalizedPicture.objects.all()
im = Image.new('RGB', (1440,num_days), background_color)
Canvas = ImageDraw.Draw(im)
for picture in pics:
# Get XY
timestamp = picture.timestamp.astimezone(est)
y_timedelta = timestamp - start
y = y_timedelta.days
x = timestamp.hour * 60 + timestamp.minute
# Paint pixel
if picture.picture is not None:
color = int_to_rgb(picture.picture.median_color)
else:
color = background_color
canvas.point((x,y), fill=color)
# All done, save
im.save(filepath)

108. pics = NormalizedPicture.objects.all()
im = Image.new('RGB', (1440,num_days), background_color)
Canvas = ImageDraw.Draw(im)
for picture in pics:
# Get XY
timestamp = picture.timestamp.astimezone(est)
y_timedelta = timestamp - start
y = y_timedelta.days
x = timestamp.hour * 60 + timestamp.minute
# Paint pixel
if picture.picture is not None:
color = int_to_rgb(picture.picture.median_color)
else:
color = background_color
canvas.point((x,y), fill=color)
# All done, save
im.save(filepath)

109. The Result
http://bit.ly/ospw-28

110. What Exactly does DST Do?
http://bit.ly/ospw-29

111. Basically It Normalizes Sunrise Time
By Making Summer Sunrise Later
http://bit.ly/ospw-29

112. At The Expense Of Wider Sunset Time
Variation, Because Of Later Summer Sunset
http://bit.ly/ospw-29

113. Python makes it easy to
answer “What If?”
So...

114. Tenet #7:
Don't be afraid to ask
“What if?”, and don't be afraid
of where it takes you.

115. Presentation:
http://bit.ly/ospw-talk-pdf
http://bit.ly/ospw-talk (raw)
Code:
http://bit.ly/ospw-code
Picture Set (9.1GB):
http://bit.ly/ospw-pics
Links and more (soon):
http://intellovations.com/ospw