Is persistency on serverless even
possible?!
Pwning AWS Lambdas & GCP Cloud Functions

What we are going to achieve
• Persistency on vulnerable Lambda &
Cloud Function
• Live exfiltration of the data – without
role keys
• Nearly undetectable method
Calc during security conferences is always something big
So here it is!

Why this topic

Yuval Avrahami (Unit 42) security research
Source: Gaining Persistency on Vulnerable Lambdas - https://unit42.paloaltonetworks.com/gaining-persistency-vulnerable-lambdas/

FaaS usage
Source: Flexera State of the Cloud Report 2022 - https://info.flexera.com/CM-REPORT-State-of-the-Cloud

What if there is vulnerability?

What if – RCE?

What if – RCE?
• Typosquatting
• Dependency confusion
• Account takeover
• Attacks on package manager
itself

Source: https://portswigger.net/daily-swig/malicious-python-library-ctx-removed-from-pypi-repo
25 May 2022
What if – RCE?

Everything is normal

Still normal

But what if?

Data gone!

How the serverless infrastructure
works

Cold Start / Cold Boot

Cold Start / Cold Boot
• First execution is slower

Cold Start / Cold Boot
• First execution is slower
• Consecutive calls are faster

Cold Start / Cold Boot
• First execution is slower
• Consecutive calls are faster
• Filesystem persists, but is frozen between calls

Cold Start - how does it works?
R e q u e s t t o
F u n c t i o n
U S E R / S E R V I C E

Cold Start - how does it works?
R e q u e s t t o
F u n c t i o n
„ H e y , w a k e u p ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E

Cold Start - how does it works?
S e t u p M i c r o V M
F I R E C R A C K E R
R e q u e s t t o
F u n c t i o n
„ H e y , w a k e u p ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E

Cold Start - how does it works?
E x e c u t e f u n c t i o n
L A M B D A
F I R E C R A C K E R
R e q u e s t t o
F u n c t i o n
„ H e y , w a k e u p ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E
S e t u p M i c r o V M

Cold Start - how does it works?
L A M B D A
F I R E C R A C K E R
R e q u e s t t o
F u n c t i o n
„ H e y , w a k e u p ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E
R e s p o n s e
E x e c u t e f u n c t i o n
S e t u p M i c r o V M

Consecutive calls
L A M B D A
F I R E C R A C K E R
R e q u e s t t o
F u n c t i o n
„ I ’ v e g o t
r e q u e s t ! ”
I N V O K E S E R V I C E
U S E R / S E R V I C E
R e s p o n s e
E x e c u t e f u n c t i o n
„ E x e c u t e t h i s ”

Consecutive calls
L A M B D A
F I R E C R A C K E R
„ I ’ v e g o t
r e q u e s t ! ”
R e q u e s t t o
F u n c t i o n
I N V O K E S E R V I C E
U S E R / S E R V I C E
R e s p o n s e
„ E x e c u t e t h i s ” E x e c u t e f u n c t i o n

GCP?

GCP?
• Nearly the same
• Containerization with gVisor (container sandbox)

Control process

AWS - Normal flow

AWS - Normal flow

GCP - Normal flow

GCP - Normal flow

Why persistency is possible in this
semi-volatile environment

Why?

Why?
• Non-volatile filesystem (at least for few minutes)

Why?
• Non-volatile filesystem (at least for few minutes)
• Writable filesystem (/tmp and memory)

Why?
• Non-volatile filesystem (at least for few minutes)
• Writable filesystem (/tmp and memory)
• Control process inside the Container / VM

AWS - Malicious flow

AWS - Malicious flow

AWS - Malicious flow

AWS - Malicious flow

GCP?

Graphic design is my passion
Monkey-patching!

GCP – Malicious flow

GCP – Malicious flow

GCP – Malicious flow

Exploitation
And demo!

Vulnerable function - CVE-2017-18342
PyYAML RCE

https://vimeo.com/user149155597

Switcher – the whole magic

Switcher – read new runtime

Switcher – save the runtime

Switcher – RAM is an option

Switcher – obtain invoke-id

Switcher – run the runtime!

Evil Bootstrap – invoke runtime

Evil Bootstrap – request handling

Evil Bootstrap – request handling this one line

Evil Bootstrap – exfiltrate!

https://vimeo.com/user149155597

Switcher – you know the drill

Switcher – get the user function

Switcher – import functions

Switcher – add to path

Switcher – exec modules

Switcher – exec modules

Switcher – monkey patch!

Evil function

Evil function – add exfiltration

Evil function – sockets!

You could try it too!
https://github.com/
Djkusik/
serverless_persistency_poc

From external process?

Mitigations & Detection

AWS GCP
Mitigations

• VPC • VPC
AWS GCP
Mitigations

• VPC
• without NAT Gateway
• VPC
AWS GCP
Mitigations

• VPC
• without NAT Gateway
• VPC
• with whole traffic routing,
without NAT Gateway
AWS GCP
Mitigations

• VPC
• without NAT Gateway
• with NAT Gateway &
Security Group not
allowing Outbound
(implicit deny)
• VPC
• with whole traffic routing,
without NAT Gateway
AWS GCP
Mitigations

• VPC
• without NAT Gateway
• with NAT Gateway &
Security Group not
allowing Outbound
(implicit deny)
• VPC
• with whole traffic routing,
without NAT Gateway
• with NAT Gateway &
Firewall rule deny whole
egress (explicit deny
required)
AWS GCP
Mitigations

AWS

AWS

GCP

GCP

It is still an RCE / Command Injection
We can find a way to evade these protections

Detection

Detection

Detection
Few ideas:
• Alerts based on execution time

Detection
Few ideas:
• Alerts based on execution time
• SDLC & CI/CD

Detection
Few ideas:
• Alerts based on execution time
• SDLC & CI/CD
• Implementing defensive layers (?)

Detection
Few ideas:
• Alerts based on execution time
• SDLC & CI/CD
• Implementing defensive layers (?)
• Flow logs (?)

How to research?

How?
• AWS Documentation

How?
• AWS Documentation
• AWS Re:Invented & Blog

How?
• AWS Documentation
• AWS Re:Invented & Blog
• Google it! (or DuckDuckGo it!)

How?
• AWS Documentation
• AWS Re:Invented & Blog
• Google it! (or DuckDuckGo it!)
• Deploy local Lambda environment (GitHub)

How?
• AWS Documentation
• AWS Re:Invented & Blog
• Google it! (or DuckDuckGo it!)
• Deploy local Lambda environment (GitHub)
• Try on real environment ;)

Lambda sHell

Here it is!
https://github.com/
Djkusik/Lambda-sHell
Based on Yuval’s SPLASH

Paweł Kusiński
Senior IT Security Consultant at Securing
Practical AWS Security trainer
@_pkusik pkusik
Thank you! Q&A time