Understanding Python garbage collection

Knowing your garbage collector
Francisco Fernandez Castano
upclose.me
francisco.fernandez.castano@gmail.com @fcofdezc
April 17, 2015
Francisco Fernandez Castano (@fcofdezc) Python GC April 17, 2015 1 / 61

Overview
1 Introduction
Motivation
Concepts
2 Algorithms
CPython RC
PyPy

Motivation
Managing memory manually is hard.
Who owns the memory?
Should I free these resources?
What happens with double frees?

Dangling pointers
int *func(void)
{
int num = 1234;
/* ... */
return &num;
}

Ownership
int *func(void)
{
int *num = malloc (10 * sizeof(int ));;
/* ... */
return num;
}

John Maccarthy

Basic concepts
Heap
A data structure in which objects may be allocated or deallocated in any
order.

Basic concepts
Heap
order.
Mutator
The part of a running program which executes application code.

Basic concepts
Heap
order.
Mutator
The part of a running program which executes application code.
Collector
The part of a running program responsible of garbage collection.

Garbage collection
Deﬁnition
Garbage collection is automatic memory management. While the
mutator runs , it routinely allocates memory from the heap. If more
memory than available is needed, the collector reclaims unused memory
and returns it to the heap.

CPython GC
CPython implementation has garbage collection.
CPython GC algorithm is Reference counting with cycle detector
It also has a generational GC.

Reference Counting Algorithm
typedef struct _object {
_PyObject_HEAD_EXTRA
Py_ssize_t ob_refcnt;
struct _typeobject *ob_type;
} PyObject;

my_list = []
class A(object ): pass
a = A()
my_list.append(a)

static int
ins1(PyListObject *self , Py_ssize_t where , PyObject
{
.
.
.
items = self ->ob_item;
for (i = n; --i >= where; )
items[i+1] = items[i];
Py_INCREF(v);
items[where] = v;
return 0;
}

#define Py_INCREF(op) (
_Py_INC_REFTOTAL _Py_REF_DEBUG_COMMA
(( PyObject *)(op))-> ob_refcnt ++)

my_list [0] = 1

#define Py_DECREF(op)

do {

if ( _Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA

--(( PyObject *)(op))-> ob_refcnt != 0)

_Py_CHECK_REFCNT (op)

else

_Py_Dealloc (( PyObject *)(op));

} while (0)

Cycles
l = []
l.append(l)
del l

Cycles

PyGC Head
typedef union _gc_head {
struct {
union _gc_head *gc_next;
union _gc_head *gc_prev;
Py_ssize_t gc_refs;
} gc;
double dummy; /* force worst -case alignment */
} PyGC_Head;

Cycles

Cycles
Wait!!

Cycles
Finalizers
Weak references

Cycles
Objects reachable from ﬁnalizers can’t safely be deleted.

Cycles
So, they’re moved to uncollectable set.

Cycles
Only reachable weakref callbacks are invoked.

CPython Memory Allocator

Demo

Reference counting
Pros: Is incremental, as it works, it frees memory.
Cons: Detecting Cycles could be hard.
Cons: Size overhead on objects.

PyPy

PyPy GC
Agnostic GC
Diﬀerent implementations over time
Nowadays it uses incminmark

Young objects
[elem * 2 for elem in elements]
balance = (a / b / c) * 4
’asdadsasd -xxx’.replace(’x’, ’y’). replace(’a’, ’
foo.bar()

PyPy memory model

PyPy GC
Minor and Major collection
Objects are moved only once
Major collection is done incrementally (to avoid long stops)

PyPy memory model

Mark and Sweep Algorithm

Mark and sweep
Pros: Can collect cycles.
Cons: Basic implementation stops the world

EuroPython

Questions?

The End

Understanding Python garbage collection

Recommended

Recommended

More Related Content

What's hot

What's hot (8)

More from fcofdezc

More from fcofdezc (9)

Recently uploaded

Recently uploaded (20)

Understanding Python garbage collection