for Singapore Ruby Meetup

1. File Upload 2015
@choonkeat
choonkeat.com
jollygoodcode.com

2. How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#"
}

3. How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#",
medium: "320x>"
}

4. How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#",
medium: "320x>",
large: "1024x>"
}

5. How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#",
medium: "320x>",
big: "640x>",
large: "1024x>"
}
Names starting to lose
meaning…

6. How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#", thumb2x: "200x200#",
medium: "320x>", medium2x: "640x>",
big: "640x>", big2x: "1280x>",
large: "1024x>", large2x: "2048x>"
}

7. How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#", thumb2x: "200x200#",
medium: "320x>", medium2x: "640x>",
big: "640x>", big2x: "1280x>",
large: "1024x>", large2x: "2048x>"
}
Reprocess all the production files, each
time, we make changes.
404 while rake runs or do at midnight?

8. How’s that transformation
juggling doing for you?

9. How’s that transformation
juggling doing for you?
class MyUploader < CarrierWave::Uploader::Base
version :thumb do
process resize_to_fill: [280, 280]
end
version :small_thumb, from_version: :thumb do
process resize_to_fill: [20, 20]
end
end

10. How’s that transformation
juggling doing for you?
class MyUploader < CarrierWave::Uploader::Base
version :thumb do
process resize_to_fill: [280, 280]
end
version :small_thumb, from_version: :thumb do
process resize_to_fill: [20, 20]
end
end
Did your users wait in the foreground
or background?

11. How’s that file path config
doing for you?

12. How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
has_attached_file :image,
url: '/system/:class/:attachment/:id/:hash-:style.:extension',
hash_secret: Rails.application.secrets.paperclip
end

13. How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
has_attached_file :image,
url: '/system/:class/:attachment/:id/:hash-:style.:extension',
hash_secret: Rails.application.secrets.paperclip
end
You sure this is the format?
Or will they need to change?

14. How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
has_attached_file :image,
url: '/system/:class/:attachment/:id/:hash-:style.:extension',
hash_secret: Rails.application.secrets.paperclip
end

15. How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
has_attached_file :image,
url: '/system/:class/:attachment/:id/:hash-:style.:extension',
hash_secret: Rails.application.secrets.paperclip
end

16. How’s that file path config
doing for you?
CarrierWave.configure do |config|
config.permissions = 0666
config.directory_permissions = 0777
config.storage = :file
end

17. How’s that file path config
doing for you?
CarrierWave.configure do |config|
config.permissions = 0666
config.directory_permissions = 0777
config.storage = :file
end
Did you configure your MySQL
data file in your app too?

18. How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
self.table = {
name: "avatars",
data: "/var/lib/mysql/data/avatars.MYD",
index: "/var/lib/mysql/data/avatars.MYI"
}
end
Did you configure your MySQL
data file in your app too?

19. How’s that form validation
error dance doing for you?

20. How’s that form validation
error dance doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Validation error: “Username is already taken!”
• Re-render form

21. How’s that form validation
error dance doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Validation error: “Username is already taken!”
• Re-render form
Where dat file go?

22. How’s that form validation
error dance doing for you?
http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err
Closed: Won’t Fix

23. How’s that form validation
error dance doing for you?
http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err

24. How’s that form validation
error dance doing for you?
http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err
Answer: use CarrierWave

25. How’s that form validation
error dance doing for you?

26. How’s that form validation
error dance doing for you?
“Is there a standard
approach? This seems like
a very common use case.”

27. How’s the schema pollution
doing for you?

28. How’s the schema pollution
doing for you?
class StoreMetadata < ActiveRecord::Migration
def change
add_column :users, :profile_image_filename, :string
add_column :users, :profile_image_size, :integer
add_column :users, :profile_image_content_type, :string
end
end

29. How’s the schema pollution
doing for you?
class StoreMetadata < ActiveRecord::Migration
def change
add_column :users, :profile_image_filename, :string
add_column :users, :profile_image_size, :integer
add_column :users, :profile_image_content_type, :string
end
end

30. How’s that multiple files
doing for you?

31. How’s that multiple files
doing for you?
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
end

32. How’s that multiple files
doing for you?
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end

33. How’s that multiple files
doing for you?
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end

34. How’s that multiple files
doing for you?
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end
Is this what you want or just
what you’re accustomed to?

35. How’s Amazon Lambda
doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Success!
• Render page with thumbnail…
How many thumbnails -
404?

36. How’s Amazon Lambda
doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Success!
• Render page with thumbnail…
Direct upload to AWS?
Cancel form submit -
delete files & thumbnails?
Deep integration &
assumption

37. How’s Dragonfly
doing for you?
http://markevans.github.io/dragonfly/

38. How’s refile
doing for you?
https://github.com/refile/refile

39. http://thecooperreview.com/10-tricks-appear-smart-meetings/
10 Tricks to Appear Smart
During Meetings

40. Take a step back

41. Take a step back
• We want to store a bunch of attributes in a model
• e.g. Title, Body, Tags, Photo

42. Take a step back
{photo}
{title}
{body}

43. Take a step back
<img src={photo}>
<h1>{title}</h1>
{body}

44. Take a step back
• Why should photo be a disproportionately
complicated attribute in my Article model?
• stored file path
• conversion
• background job
• aws config
• clean up on delete

45. Take a step back
• Why should photo be a disproportionately
complicated attribute in my Article model?
• stored file path
• conversion
• background job
• validation error dance
• aws config

46. Take a step back
• Frankly photo_url is best; least intrusive

47. Take a step back
• Frankly photo_url is best; least intrusive
• Problems
• Remote url 404? (not exclusive to your app)
• Asking users to give us a URL is a hard sell
• Need to render other sizes
• Filter by meta data

48. Take a step back
• Frankly photo_url is best; least intrusive
• Problems
• Remote url 404? (not exclusive to your app)
• Asking users to give us a URL is a hard sell
• Need to render other sizes
• Filter by meta data

49. Take a step back
• Frankly photo_url is best; least intrusive
• Problems
• Remote url 404? (not exclusive to your app)
• Asking users to give us a URL is a hard sell
• Need to render other sizes
• Filter by meta data

50. Take a step back
• Frankly photo_url is best; least intrusive
• Solutions
• Exclusive server for your app
• Upload to that server
• On-the-fly resize based on URL
• Store url with meta data: photo_json instead?

51. Just add server

52. • PostgreSQL, MySQL
• Redis
• Memcached
• SMTP server (Mail)
You are already
Generic server to do
specialised work
Not specific to your
business logic

53. • Not a new pattern
• Mostly commercial services
• Maybe it has to be Free & Open Source to become
a default pattern
Image server

54. Want
• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist

55. Want
• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
What would my app look
like in a better world?

56. My app: Bare minimum
create_table "users" do |t|
t.string "name"
t.json "avatar"
t.json "photos"
end

57. My app: Bare minimum
create_table "users" do |t|
t.string "name"
t.json "avatar"
t.json "photos" # multiple files in a column
end

58. My app: Bare minimum
Image serverRails appBrowser
{ avatar: #<File..> }

59. My app: Bare minimum
Image serverRails appBrowser
{ avatar: #<File..> }
{“path”:“x.jpg”,
“geometry”:“200x600”}
#<File..>

60. My app: Bare minimum
Image serverRails appBrowser
{“path”:“x.jpg”,
“geometry”:“200x600”}
#<File..>
user.avatar={“path”: “x.jpg”…}
user.save
<img src=“x.jpg”>
{ avatar: #<File..> }

61. My app: Bare minimum
Image serverRails appBrowser
{“path”:“x.jpg”,
“geometry”:“200x600”}
<img src=“x.jpg”>
#<File..>
GET x.jpg
#<File..>
user.avatar={“path”: “x.jpg”…}
user.save
{ avatar: #<File..> }

62. My app: Bare minimum
• Browser render <file> field; regular form submit
• Receive binary param, uploads to attache server
and stores the json response instead
• Your app render <img src> requesting for image in
certain size, e.g. http://example/200x/
file.png

63. Image serverRails appBrowser
#<File..>
{“path”:“x.jpg”,
“geometry”:“200x600”}
Progressive Enhancement

64. Image serverRails appBrowser
#<File..>
{“path”:“x.jpg”,
“geometry”:“200x600”}
Progressive Enhancement
{ avatar: {“path”:“x.jpg”, … }

65. { avatar: {“path”:“x.jpg”, … }
Image serverRails appBrowser
#<File..>
{“path”:“x.jpg”,
“geometry”:“200x600”}
user.avatar={“path”: “x.jpg”…}
user.save
<img src=“x.jpg”>
Progressive Enhancement

66. • Browser render <file> field; regular form submit
• Receive binary param, uploads to attache server
and stores the json response instead
• Your app render <img src> requesting for image in
certain size, e.g. http://example/200x/
file.png
Progressive Enhancement

67. • Browser render <file> field; regular form submit
• Receive binary param, uploads to attache server
and stores the json response instead
• Your app render <img src> requesting for image in
certain size, e.g. http://example/200x/
file.png
• JS upload directly to attache server; “Direct upload” in
AWS parlance
• No binary in params; receive and store json attribute
Progressive Enhancement
• When after_update & after_destroy
removes obsolete file from attache via delete API

68. • Just use Ruby; just use your framework
• Pre-render multiple sizes
• fetch the urls, server will generate and cache
• Validation
• validating a regular json attribute
How do I…

69. • Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
Want (cont’d)

70. Want (cont’d)
• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
• Separate, standalone server
• Minimal / zero ongoing administration

71. Want (cont’d)
• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
• Separate, standalone server
• Minimal / zero ongoing administration
How does this
server work?

72. Attache File Server
• HTTP server with simple APIs
• upload
• download
• delete
• Rack app + ImageMagick
• Go? Node? C++? PHP?
• GraphicsMagick? MyResizer.bash?

73. • Uploaded files are stored locally
• Resize local file on-the-fly
• configurable pool size to limit concurrent resizing
• Sync upload to cloud storage
• 2 hop problem vs complexity
• Fixed local storage, purge LRU (zero maintenance)
• Spin up new fresh servers anytime…
Attache File Server

74. • When requested file does not exist locally
• fetch from cloud storage & write locally
• resume operations
Attache File Server

75. • Remove obsolete file is “best effort”
• If photo delete failed, do you Error 500 or stop the
world?
• OCDs can schedule rake task remove dangling files?
Attache File Server

76. • Caching in production
• Browser → CDN → Varnish → Disk → Cloud
Attache File Server

77. Demo
https://attache-demo.herokuapp.com/

78. https://github.com/choonkeat/attache_api
ATTACHE_URL=http://localhost:9292
ATTACHE_SECRET_KEY=topsecret
rake
Compatibility Check

79. tus.io
• Open protocol for resumable uploads built on HTTP
• Perfect for mobile apps
• Rack middleware implemented in choonkeat/
attache#10

80. Responsive Images with
Client Hints
http://blog.imgix.com/2015/10/13/next-generation-responsive-images-with-client.html

81. SmartCrop
https://github.com/jwagner/smartcrop.js/

82. Dragonfly & refile

83. Dragonfly & refile
• tldr: we can fuss over implementation, but it is
mostly about architecture

84. Dragonfly & refile
• Rack middleware in your Rails app by default
• performing on-the-fly image resize 😱
• Rack standalone end point
• Dragonfly.app - upload, download, delete
• Refile::App - upload, download, delete
• Downloads are unthrottled

85. Dragonfly & refile
• BEFORE: Rails integrate with AWS
• AFTER: Rails integrate with AWS + Rack app
• Maintain identical AWS config in both apps
• Rails app couldn’t shed the “concern”
• Multiple images still require multiple models

86. refile
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
accepts_attachments_for :images, attachment: :file
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end

87. refile
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
accepts_attachments_for :images, attachment: :file
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end
“Note it must be possible to persist images
given only the associated post and a file. There
must not be any other validations or constraints
which prevent images from being saved”
i.e. Must be pure dummy wrapper; no
validations here

88. refile download
https://github.com/refile/refile/blob/master/lib/refile/app.rb
get "/:token/:backend/:processor/:id/:filename" do
halt 404 unless download_allowed?
stream_file processor.call(file)
end

89. refile download
https://github.com/refile/refile/blob/master/lib/refile/app.rb
get "/:token/:backend/:processor/:id/:filename" do
halt 404 unless download_allowed?
stream_file processor.call(file)
end
How many ImageMagick can
you run in parallel?
has_many :images?

90. refile upload
https://github.com/refile/refile/blob/master/lib/refile/app.rb
post "/:backend" do
halt 404 unless upload_allowed?
tempfile = request.params.fetch("file").fetch(:tempfile)
file = backend.upload(tempfile)
content_type :json
{ id: file.id }.to_json
end
def file
file = backend.get(params[:id])
unless file.exists?
log_error("Could not find attachment by id: #{params[:id]}")
halt 404
end
file.download
end

91. post "/:backend" do
halt 404 unless upload_allowed?
tempfile = request.params.fetch("file").fetch(:tempfile)
file = backend.upload(tempfile)
content_type :json
{ id: file.id }.to_json
end
def file
file = backend.get(params[:id])
unless file.exists?
log_error("Could not find attachment by id: #{params[:id]}")
halt 404
end
file.download
end
refile upload
https://github.com/refile/refile/blob/master/lib/refile/app.rb
2 hops problem when
uploading and downloading
• 3mb file becomes 6mb each
way
• #create and #show
becomes 12mb process
• has_many :images?

92. refile
• CarrierWave-styled named processors (e.g. :fill, :thumb) vs
passing through syntax to underlying ImageMagick
• personally prefer leveraging off existing knowledge
• instead of manually configured syntax sugar
• “2 hop problem” however provide higher consistency
when running multiple refile servers
• upload-here-download-there problem
• (considering to perform 2-hop upload instead of async)

93. refile
• Can upload to S3 direct and/or upload to refile
• Redundancy interesting, but prefer less concern in Rails
app
• Concept of cache and store to manage “dangling file
problem” is worth considering
• Download urls are signed-only (this practice should be
adopted)
• can partly counter motivation to abuse “dangling file
problem” (aka free file hosting, whee!)

94. Questions?
Attache server
https://github.com/choonkeat/attache
Gem for Rails
https://github.com/choonkeat/attache_rails
Demo
https://attache-demo.herokuapp.com/