Everything I know about S3 pre-signed URLs

AWS User Group: Dublin Meetup / 2022-11-17
1

META_SLIDE!
fth.link/presign
loige 2

😎"I have a startup idea..."
loige
fth.link/presign
3

loige
meower
fth.link/presign
4

Yes, I still have to implement the
proﬁle picture upload feature... 😅
loige 9

$ ~ whoami
👋I'm Luciano ( 🍕🍝 )
Senior Architect @ fourTheorem (Dublin )
nodejsdp.link
📔Co-Author of Node.js Design Patterns 👉
Let's connect!
(blog)
(twitter)
(twitch)
(github)
loige.co
@loige
loige
lmammino 10

Always re-imagining
We are a pioneering technology consultancy
focused on AWS and serverless
| |
Accelerated Serverless AI as a Service Platform Modernisation
loige
✉Reach out to us at
😇We are always looking for talent:
hello@fourTheorem.com
fth.link/careers
11

We host a weekly podcast about AWS
awsbites.com
loige 12

🤔
"upload" feature
in a web app...
loige 13

What's an upload, really?
loige 14

loige
Structure of an HTTP request
POST /profilepic/upload HTTP/1.1
Host: api.meower.com
Content-Type: text/plain
Content-Length: 9
Some data
Method
Path Version
Headers
Body
16

loige
What if it's a binary (like a picture)?
PUT /profilepic/upload HTTP/1.1
Host: api.meower.com
Content-Type: image/jpeg
Content-Length: 2097852
��JFIFHH��"��
��Dl��FW�'6N�()H�'p��FD3 [...]
read 2097852
bytes
17

loige
Using Lambda
Lambda proxy integration*
* JSON Based protocol mapping to HTTP
(JSON Request / JSON Response)
18

{
"resource": "/profilepic/upload",
"path": "/profilepic/upload",
"httpMethod": "POST",
"headers": {
"Content-Type": "text/plain",
"Content-Length": "9",
"Host": "api.meower.com",
},
"body": "Some data"
}
1
2
3
4
5
6
7
8
9
10
11
{
1
2
3
4
"headers": {
5
6
7
8
},
9
"body": "Some data"
10
}
11
{
1
2
3
4
"headers": {
5
6
7
8
},
9
"body": "Some data"
10
}
11
"headers": {
},
{
1
2
3
4
5
6
7
8
9
"body": "Some data"
10
}
11
"body": "Some data"
{
1
2
3
4
"headers": {
5
6
7
8
},
9
10
}
11
{
"headers": {
},
"body": "Some data"
}
1
2
3
4
5
6
7
8
9
10
11
loige
Lambda proxy integration (request)
19

{
"httpMethod": "PUT",
"headers": {
"Content-Type": "image/jpeg",
},
"body": "????????"
}
1
2
3
4
5
6
7
8
9
10
11
{
1
2
3
4
"headers": {
5
6
7
8
},
9
"body": "????????"
10
}
11
"body": "????????"
{
1
2
3
4
"headers": {
5
6
7
8
},
9
10
}
11
{
"headers": {
},
"body": "????????"
}
1
2
3
4
5
6
7
8
9
10
11
loige
Lambda proxy integration (request - picture)
20

"isBase64Encoded": true,
"body": "/9j/4AAQSkZJRgABAQEASABIAAD/2w[...]"
{
1
2
3
4
"headers": {
5
6
7
8
},
9
10
11
}
12
{
"headers": {
},
"isBase64Encoded": true,
"body": "/9j/4AAQSkZJRgABAQEASABIAAD/2w[...]"
}
1
2
3
4
5
6
7
8
9
10
11
12
loige
Lambda proxy integration (request - picture)
21

loige
1. Parse request (JSON)
2. Decode body (Base64)
3. Validation / resize
4. ...
Lambda proxy integration request
/proﬁlepic/upload
22

loige
1. Parse request (JSON)
2. Decode body (Base64)
3. Validation / resize
4. ...
Lambda proxy integration request
/proﬁlepic/upload
😎 23

loige
Is this a good solution? 🙄
24

loige
Limitations...
API Gateway requests timeout: 30 sec
API Gateway payload: max 10 MB
Lambda timeout: max 15 mins
Lambda payload size: max 6 MB
Upload: 6 MB / 30 sec
25

loige
... not really!
What about supporting big images or even videos?
26

loige
An alternative approach
✅ Long lived connection
✅ No size limit
27

loige
S3 pre-signed URLs 😱
An S3 built-in feature
to authorize operations (download, upload, etc) on a
bucket / object
using time-limited authenticated URLs
29

loige
Using S3 pre-signed URLs for upload
* yeah, this can be a Lambda as well 😇
*
30

loige
*
31

loige
*
32

loige
*
33

loige
✅
*
34

loige
... and we can also use it for downloads! 🤩
35

loige
Using S3 pre-signed URLs for download
36

loige
37

loige
38

loige
39

loige
✅
40

loige
⚠VERY important details!
I lied to you a little in those diagrams... 🤥
It's a decent mental model, but it's not accurate 😅
The server never really talks with S3!
The server actually creates the signed URL by itself!
We will see later what's the security model around this idea!
41

loige
✅It's a managed feature (a.k.a. no servers to manage)
✅We can upload and download arbitrarily big ﬁles with no practical limits*
✅Reasonably simple and secure
👍Seems good to me!
* objects in S3 are "limited" to 5TB (when using multi-part upload), 5 GB otherwise.
42

loige
Generating our ﬁrst pre-signed URL
$ aws s3 presign
s3://finance-department-bucket/2022/tax-certificate.pdf
https://s3.amazonaws.com/finance-department-bucket/2022/tax-
certificate.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-
Credential=AKIA3SGQVQG7FGA6KKA6%2F20221104%2Fus-east-
1%2Fs3%2Faws4_request&X-Amz-Date=20221104T140227Z&X-Amz-
Expires=3600&X-Amz-SignedHeaders=host&X-Amz-
Signature=b228dbec8c1008c80c162e1210e4503dceead1e4d4751b4d9787
314fd6da4d55
Whoever has this URL can
download the tax certiﬁcate!
43

loige
What's in a pre-signed URL
https://s3.amazonaws.com/finance-department-bucket/2022/tax-
certificate.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-
Credential=AKIA3SGQVQG7FGA6KKA6%2F20221104%2Fus-east-
Signature=b228dbec8c1008c80c162e1210e4503dceead1e4d4751b4d9787
314fd6da4d55
44

loige
What's in a pre-signed URL
https://s3.amazonaws.com
/ﬁnance-department-bucket
/2022/tax-certiﬁcate.pdf
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIA3SGQXQG7XXXYKKA6%2F20221104...
&X-Amz-Date=20221104T140227Z
&X-Amz-Expires=3600
&X-Amz-SignedHeaders=host
&X-Amz-Signature=b228dbec8c1008c80c162e1210e4503dceead1e4d4...
What if I change this to
/passwords.txt?
45

loige
Pre-signed URLs validation
https://s3.amazonaws.com
/ﬁnance-department-bucket
/2022/tax-certiﬁcate.pdf
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIA3SGQXQG7XXXYKKA6%2F20221104...
&X-Amz-Date=20221104T140227Z
&X-Amz-Expires=3600
&X-Amz-SignedHeaders=host
&X-Amz-Signature=b228dbec8c1008c80c162e1210e4503dceead1e4d4...
47

loige
🤓
Once a pre-signed URL is generated you
cannot edit it without breaking it
Photo by on
CHUTTERSNAP Unsplash
⚠Also note that you can use a pre-signed URL as many
times as you want until it expires
48

loige
🔐Permissions
Anyone with valid credentials can create a pre-signed URL (client side)
valid credentials = Role, User, or Security Token
The generated URL inherits the permissions of the credentials used to generate it
This means you can generate pre-signed URLs for things you don't have access to 😅
49

loige
$ aws s3 presign s3://ireland/i-love-you
https://ireland.s3.eu-west-1.amazonaws.com/i-love-you?X-Amz-
Algorithm=AWS4-HMAC-SHA256&X-Amz-
Credential=AKIA3ABCVQG7FGA6KKA6%2F20221115%2Feu-west-
Signature=75749c92d94d03e411e7bbf64419f2af09301d1791b0df54c639
137c715f7888
😱
I swear I don't even know if this
bucket exists or who owns it!
50

loige
Pre-signed URLs are validated at request time
51

loige
Creating a pre-signed URL
programmatically
AWS SDK for JavaScript v3
52

import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'
const s3Client = new S3Client()
const command = new GetObjectCommand({
Bucket: "some-bucket",
Key: "some-object"
})
const preSignedUrl = await getSignedUrl(s3Client, command, {
expiresIn: 3600
})
console.log(preSignedUrl)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1
2
3
4
5
6
7
Key: "some-object"
8
})
9
10
11
expiresIn: 3600
12
})
13
14
15
1
2
3
4
5
6
7
Key: "some-object"
8
})
9
10
11
expiresIn: 3600
12
})
13
14
15
Key: "some-object"
})
1
2
3
4
5
6
7
8
9
10
11
expiresIn: 3600
12
})
13
14
15
expiresIn: 3600
})
1
2
3
4
5
6
7
Key: "some-object"
8
})
9
10
11
12
13
14
15 console.log(preSignedUrl)
1
2
3
4
5
6
7
Key: "some-object"
8
})
9
10
11
expiresIn: 3600
12
})
13
14
15
Key: "some-object"
})
expiresIn: 3600
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
loige 53

loige
📦
Uploading a ﬁle
using pre-signed URLs
54

loige
2 Options: PUT & POST 🤨
55

loige
PUT Method
PUT <preSignedURL> HTTP/1.1
Host: <bucket>.s3.<region>.amazonaws.com
��Dl��FW�'6N�()H�'p��FD3 [...]
56

const command = new PutObjectCommand({
Key: "some-object"
})
expiresIn: 3600
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
loige
Only diﬀerence with the
previous example
57

loige
PUT Method - Limitations
You cannot set a limit on the upload size (max of 5 GB)! *
You can limit the Content-Type but you can specify exactly one
* Unless you know the exact size in advance
58

loige
POST method
It uses the multipart/form-data encoding (form upload)
Gives more freedom to the client to shape the request (Content-Type, file name, etc)
It uses a policy mechanism to define the "rules" of what can be uploaded
E.g. you can limit the supported mime types and provide a maximum file size
You can use it to upload from a web form and even configure the redirect URL
It's not really a URL but more of a pre-signed form!
59

POST / HTTP/1.1
Host: <bucket>.s3.amazonaws.com
Content-Type: multipart/form-data; boundary=9431149156168
--9431149156168
Content-Disposition: form-data; name="key"
picture.jpg
--9431149156168
Content-Disposition: form-data; name="X-Amz-Credential"
AKIA3SGABCDXXXA6KKA6/20221115/eu-west-1/s3/aws4_request
--9431149156168
Content-Disposition: form-data; name="Policy"
eyJleHBpcmF0aW9uIjoiMjAyMi0xMS0xNVQyMDo0NjozN1oiLCJjb25kaXRpb25zIjpbWyJj[...]
--9431149156168
Content-Disposition: form-data; name="X-Amz-Signature"
2c1da0001dfec7caea1c9fb80c7bc8847f515a9e4483d2942464f48d2f827de7
--9431149156168
Content-Disposition: form-data; name="file"; filename="MyFilename.jpg"
Content-Type: image/jpeg
��Dl��FW�'6N�()H�'p��FD3[...]
--9431149156168--
loige
60

loige
POST method Policy
A JSON object (Base64 encoded) that deﬁnes the upload rules (conditions) and the
expiration date
This is what gets signed: you cannot alter the policy without breaking the signature
{
"expiration": "2022-11-15T20:46:37Z",
"conditions": [
["content-length-range", 0, 5242880],
["starts-with", "$Content-Type", "image/"],
{"bucket": "somebucket"},
{"X-Amz-Algorithm": "AWS4-HMAC-SHA256"},
{"X-Amz-Credential": "AKIA3SGABCDXXXA6KKA6/20221115/eu-west-1/s3/aws4_request"},
{"X-Amz-Date": "20221115T194637Z"},
{"key": "picture.jpg"}
]
}
61

import { S3Client } from '@aws-sdk/client-s3'
import { createPresignedPost } from '@aws-sdk/s3-presigned-post'
const { BUCKET_NAME, OBJECT_KEY } = process.env
const { url, fields } = await createPresignedPost(s3Client, {
Bucket: 'somebucket',
Key: 'someobject',
Conditions: [
['content-length-range', 0, 5 * 1024 * 1024] // 5 MB max
],
Fields: {
success_action_status: '201',
'Content-Type': 'image/png'
},
Expires: 3600
})
console.log({ url, fields })
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1
2
3
4
5
6
7
8
Key: 'someobject',
9
Conditions: [
10
11
],
12
Fields: {
13
14
15
},
16
Expires: 3600
17
})
18
19
20
})
1
2
3
4
5
6
7
8
Key: 'someobject',
9
Conditions: [
10
11
],
12
Fields: {
13
14
15
},
16
Expires: 3600
17
18
19
20
1
2
3
4
5
6
7
8
Key: 'someobject',
9
Conditions: [
10
11
],
12
Fields: {
13
14
15
},
16
Expires: 3600
17
})
18
19
20
Key: 'someobject',
1
2
3
4
5
6
7
8
9
Conditions: [
10
11
],
12
Fields: {
13
14
15
},
16
Expires: 3600
17
})
18
19
20
Conditions: [
],
1
2
3
4
5
6
7
8
Key: 'someobject',
9
10
11
12
Fields: {
13
14
15
},
16
Expires: 3600
17
})
18
19
20
Fields: {
},
1
2
3
4
5
6
7
8
Key: 'someobject',
9
Conditions: [
10
11
],
12
13
14
15
16
Expires: 3600
17
})
18
19
20
Expires: 3600
1
2
3
4
5
6
7
8
Key: 'someobject',
9
Conditions: [
10
11
],
12
Fields: {
13
14
15
},
16
17
})
18
19
20 console.log({ url, fields })
1
2
3
4
5
6
7
8
Key: 'someobject',
9
Conditions: [
10
11
],
12
Fields: {
13
14
15
},
16
Expires: 3600
17
})
18
19
20
Key: 'someobject',
Conditions: [
],
Fields: {
},
Expires: 3600
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
loige 62

// you can use `url` and `fields` to generate an HTML form
const code = `<h1>Upload an image to S3</h1>
<form action="${url}" method="post" enctype="multipart/form-data">
${Object.entries(fields).map(([key, value]) => {
return `<input type="hidden" name="${key}" value="${value.replace(/"/g, '"')}">`
}).join('n')}
<div><input type="file" name="file" accept="image/png"></div>
<div><input type="submit" value="Upload"></div>
</form>`
1
2
3
4
5
6
7
8
9
10
loige 63

loige
Limitations and quirks
It supports only 1 file (cannot upload multiple files in one go)
The file field must be the last entry in the form
(S3 will ignore every other field after the file)
From the browser (AJAX) you need to enable CORS on the bucket
64

loige
Should I use PUT or POST? 🧐
PUT is simpler but deﬁnitely more limited
POST is slightly more complicated (and less adopted) but it's more ﬂexible
You should probably put some time into learning POST and use that!
65

loige
Pre-signed URLs for other operations
S3 pre-signed URLs are not limited to GET, PUT or POST operations
You can literally create pre-signed URLs for any command
(DeleteObject, ListBuckets, MultiPartUpload, etc...)
66

loige
Do you need moar examples? 😼
github.com/lmammino/s3-presigned-urls-examples
67

loige
... In summary
S3 pre-signed URLs are a great way to authorise operations on S3
They are generally used to implement upload/download features
The signature is created client-side so you can sign anything. Access is validated at
request time
This is not the only solution, you can also use the JavaScript SDK from the frontend
and get limited credentials from Cognito (Amplify makes that process simpler)
For upload you can use PUT and POST, but POST is much more ﬂexible
💬PS: Meower.com doesn't really exist... but... do you want to invest?! It's a great
idea, trust me!
68

Cover photo by on
Kelly Sikkema Unsplash
fourtheorem.com
THANKS! 🙌
fth.link/presign
loige
It's a wrap!
69

Everything I know about S3 pre-signed URLs

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Everything I know about S3 pre-signed URLs

Ähnlich wie Everything I know about S3 pre-signed URLs (20)

Mehr von Luciano Mammino

Mehr von Luciano Mammino (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Everything I know about S3 pre-signed URLs