1. IP: 10.4.0.2
Host: graphite-02
IP: 10.4.0.1
Host: graphite-01
[cache:a]
line_receiver_interface 0.0.0.0
line_receiver_port 2103
pickle_receiver_interface 0.0.0.0
pickle_receiver_port 2104
cache_query_interface 0.0.0.0
cache_query_port 7102
[cache:b]
line_receiver_interface 0.0.0.0
line_receiver_port 2203
pickle_receiver_interface 0.0.0.0
pickle_receiver_port 2204
cache_query_interface 0.0.0.0
cache_query_port 7202
[relay]
line_receiver_interface = 0.0.0.0
line_receiver_port = 2003
pickle_receiver_interface = 0.0.0.0
pickle_receiver_port = 2004
relay_method = consistent-hashing
replication_factor = 1
Destinations = [ 10.4.0.1:2104:a,
10.4.0.1:2204:b,
10.4.0.2:2104:a,
10.4.0.2:2204:b ]
[cache:a]
line_receiver_interface 0.0.0.0
line_receiver_port 2103
pickle_receiver_interface 0.0.0.0
pickle_receiver_port 2104
cache_query_interface 0.0.0.0
cache_query_port 7102
[cache:b]
line_receiver_interface 0.0.0.0
line_receiver_port 2203
pickle_receiver_interface 0.0.0.0
pickle_receiver_port 2204
cache_query_interface 0.0.0.0
cache_query_port 7202
[relay]
line_receiver_interface = 0.0.0.0
line_receiver_port = 2003
pickle_receiver_interface = 0.0.0.0
pickle_receiver_port = 2004
relay_method = consistent-hashing
replication_factor = 1
Destinations = [ 10.4.0.1:2104:a,
10.4.0.1:2204:b,
10.4.0.2:2104:a,
10.4.0.2:2204:b ]
[aggregator]
line_receiver_interface = 0.0.0.0
line_receiver_port = 2013
pickle_receiver_interface = 0.0.0.0
pickle_receiver_port = 2014
[aggregator]
line_receiver_interface = 0.0.0.0
line_receiver_port = 2013
pickle_receiver_interface = 0.0.0.0
pickle_receiver_port = 2014
load balancer
IP: 10.4.0.10
Host: graphite
TCP ports: 2003, 2004 HTTP ports: 80
[webapp]
port = 80
memcache_hosts = [
“rf-1.cache.amazonaws.com” ]
cluster_servers = [
“10.4.0.2:80” ]
remote_rendering = false
carbonlink_hosts = [
“10.4.0.1:7102”,
“10.4.0.1:7202” ]
[webapp]
port = 80
memcache_hosts = [
“rf-1.cache.amazonaws.com” ]
cluster_servers = [
“10.4.0.2:80” ]
remote_rendering = false
carbonlink_hosts = [
“10.4.0.2:7102”,
“10.4.0.2:7202” ]
adobrosynets@recordedfuture.com

2. IP: 10.4.0.2
Host: graphite-02
IP: 10.4.0.1
Host: graphite-01
[cache:a]
line_receiver_port 2103
pickle_receiver_port 2104
cache_query_port 7102
[cache:b]
line_receiver_port 2203
pickle_receiver_port 2204
cache_query_port 7202
[relay]
line_receiver_port = 2003
pickle_receiver_port = 2004
relay_method = consistent-hash
replication_factor = 1
Destinations = [
10.4.0.1:2104:a,10.4.0.1:2204:b,
10.4.0.2:2104:a,10.4.0.2:2204:b,
10.4.0.3:2104:a,10.4.0.3:2204:b
]
[cache:a]
line_receiver_port 2103
pickle_receiver_port 2104
cache_query_port 7102
[cache:b]
line_receiver_port 2203
pickle_receiver_port 2204
cache_query_port 7202
[relay]
line_receiver_port = 2003
pickle_receiver_port = 2004
relay_method = consistent-hash
replication_factor = 1
Destinations = [
10.4.0.1:2104:a,10.4.0.1:2204:b,
10.4.0.2:2104:a,10.4.0.2:2204:b ,
10.4.0.3:2104:a,10.4.0.3:2204:b
]
[aggregator]
line_receiver_port = 2013
pickle_receiver_port = 2014
[aggregator]
line_receiver_port = 2013
pickle_receiver_port = 2014
load balancer
IP: 10.4.0.10
Host: graphite
TCP ports: 2003, 2004 HTTP ports: 80
[webapp]
memcache_hosts = [
“rf-1.cache” ]
cluster_servers = [
“10.4.0.2:80”,
“10.4.0.3:80” ]
carbonlink_hosts = [
“10.4.0.1:7102”,
“10.4.0.1:7202” ]
[webapp]
memcache_hosts = [
“rf-1.cache” ]
cluster_servers = [
“10.4.0.1:80”,
“10.4.0.3:80” ]
carbonlink_hosts = [
“10.4.0.2:7102”,
“10.4.0.2:7202” ]
IP: 10.4.0.3
Host: graphite-03
[cache:a]
line_receiver_port 2103
pickle_receiver_port 2104
cache_query_port 7102
[cache:b]
line_receiver_port 2203
pickle_receiver_port 2204
cache_query_port 7202
[relay]
line_receiver_port = 2003
pickle_receiver_port = 2004
relay_method = consistent-hash
replication_factor = 1
Destinations = [
10.4.0.1:2104:a,10.4.0.1:2204:b,
10.4.0.2:2104:a,10.4.0.2:2204:b ,
10.4.0.3:2104:a,10.4.0.3:2204:b
]
[aggregator]
line_receiver_port = 2013
pickle_receiver_port = 2014
[webapp]
memcache_hosts = [
“rf-1.cache” ]
cluster_servers = [
“10.4.0.1:80”,
“10.4.0.2:80” ]
carbonlink_hosts = [
“10.4.0.3:7102”,
“10.4.0.3:7202” ]
adobrosynets@recordedfuture.com

3. Key points
- Many nodes, each node running carbon-relay, webapp, carbon-cache(s).
- Use at least two carbon-cache processes at the node to utilize performance
(typically one process per CPU core)
- All carbon-cache instances use the same schema definitions for whisper files
- All monitoring agents (statsd/sensu/gdash/codehale/collectd/etc) use loadbalancer front-end (HAproxy or ELB)
to send/query metrics.
- Each carbon-relay may route metrics to any carbon-cache instance at any graphite server in cluster.
- All carbon-relays use 'consistent-hashing' method and have exactly the same DESTINATIONS list
(carbon.conf DESTINATIONS. Order is important?)
- All webapp processes share exactly the same memcache instance(s)
(local_settings.py MEMCACHE_HOSTS)
- Each webapp can may query only local carbon-cache instances.
(local_settings.py CARBONLINK_HOSTS)
- All webapps may contain not only other webapps in CLUSTER_SERVERS , but also itself.
(local_settings.py, as of 0.9.10 version)
- Each webapp CARBONLINK_HOSTS must contain only local instances from DESTINATIONS
(order is not important )
- In terms of AWS EC2, graphite nodes are supposed to be installed in the same Region.
- Aggregator is not that useful. It is better to aggregate somewhere else (statsd/diamond) and send to graphite.

4. webapp/graphite/storage.py
STORE = Store(settings.DATA_DIRS, remote_hosts=settings.CLUSTER_SERVERS)
class Store:
def __init__(self, directories=[], remote_hosts=[]):
self.directories = directories
self.remote_hosts = remote_hosts
self.remote_stores = [ RemoteStore(host)
for host in remote_hosts if not is_local_interface(host) ]
...
def find_first():
...
remote_requests = [ r.find(query) for r in self.remote_stores if r.available ]
...
It is safe to have exactly the same CLUSTER_SERVERS option for all webapps in a cluster
(less template work with Chef/Puppet)
Though, there are some edge-cases. https://github.com/graphite-project/graphite-web/issues/222

5. CARBONLINK_HOSTS should contain only local carbon-cache instances, not all DESTINATIONS list.
Webapp will take care of selecting proper carbon-cache instance for the metric, although it has a different
list of items in his hash ring .
https://answers.launchpad.net/graphite/+question/228472
webapp/graphite/render/datalib.py
# Data retrieval API
def fetchData(requestContext, pathExpr):
...
if requestContext['localOnly']:
store = LOCAL_STORE
else:
store = STORE
for dbFile in store.find(pathExpr):
log.metric_access(dbFile.metric_path)
dbResults = dbFile.fetch( timestamp(startTime), timestamp(endTime) )
try:
cachedResults = CarbonLink.query(dbFile.real_metric)
results = mergeResults(dbResults, cachedResults)
except:
log.exception()
results = dbResults
if not results:
continue
...
return seriesList

6. Useful links:
http://graphite.readthedocs.org
http://www.aosabook.org/en/graphite.html
http://rcrowley.org/articles/federated-graphite.html
http://bitprophet.org/blog/2013/03/07/graphite/
http://boopathi.in/blog/the-graphite-story-directi
https://answers.launchpad.net/graphite/+question/228472