Developers in open source projects must make decisions on contributions from other community members, such as whether or not to accept a pull request. However, secondary factors—beyond the code itself—can influence those decisions. For example, signals from GitHub profiles, such as a number of followers, activity, names, or gender can also be considered when developers make decisions. In this paper, we examine how developers use these signals (or not) when making decisions about code contributions. To evaluate this question, we evaluate how signals related to perceived gender identity and code quality influenced decisions on accepting pull requests. Unlike previous work, we analyze this decision process with data collected from an eye-tracker. We analyzed differences in what signals developers said are important for themselves versus what signals they actually used to make decisions about others. We found that after the code snippet (x=57%), the second place programmers spent their time fixating on supplemental technical signals(x=32%), such as previous contributions and popular repositories. Diverging from what participants reported themselves, we also found that programmers fixated on social signals more than recalled.

1. Beyond the Code Itself:
How Programmers Really Look
at Pull Requests
Denae Ford, Mahnaz Behroozi, Alexander Serebrenik, Chris ParninDenaeFordRobin
North Carolina State University + Eindhoven University Of Technology

2. PULL REQUEST: SUBMITTING CODE ONLINE 2
https://guides.github.com/introduction/flow/
Open a Pull Request

3. PULL REQUEST: SUBMITTING CODE ONLINE 2
https://guides.github.com/introduction/flow/
Open a Pull RequestPull Request Is Merged

4. PULL REQUEST: SUBMITTING CODE ONLINE 2
https://guides.github.com/introduction/flow/
Open a Pull RequestPull Request Is MergedDiscuss + Review

5. CODE CONTRIBUTIONS BASED ON “MERITS”? 3
https://help.github.com/en/articles/about-your-profile

6. CODE CONTRIBUTIONS BASED ON “MERITS”? 3
https://help.github.com/en/articles/about-your-profile

7. ABBY’S GITHUB PROFILE PAGE 4

8. ABBY’S GITHUB PROFILE PAGE 4

9. 5ABBY’S GITHUB PROFILE PAGE

10. 6BUILDING ON PRIOR WORK
Previous studies were done
post factum
Further insight into the
decision making process
Eye tracking offers a holistic
perspective to the story
[Dabbish, 2012] [Tsay, 2014] [Marlow 2013]

11. 6BUILDING ON PRIOR WORK
Previous studies were done
post factum
Further insight into the
decision making process
Eye tracking offers a holistic
perspective to the story
[Dabbish, 2012] [Tsay, 2014] [Marlow 2013]

12. 6BUILDING ON PRIOR WORK
Previous studies were done
post factum
Further insight into the
decision making process
Eye tracking offers a holistic
perspective to the story
[Dabbish, 2012] [Tsay, 2014] [Marlow 2013]

13. 6BUILDING ON PRIOR WORK
Previous studies were done
post factum
Further insight into the
decision making process
Eye tracking offers a holistic
perspective to the story
[Dabbish, 2012] [Tsay, 2014] [Marlow 2013]

14. How do programmers review pull
requests?
Where do programmers think they
look vs. where they really look?
What strategies do people use to
manage signals for their personal
identity?
7
RQ2
RQ1
RQ3
RESEARCH QUESTIONS

15. 8METHODOLOGY
PROTOCOL
Pre Experiment
Survey
Have you ever accepted or rejected or submitted a
pull request? *
Please rate your programming experience on a scale
from 1-10. *
How much experience do you have performing
integration tasks? *
Do you have a GitHub account? *

16. 9
PROTOCOL
Pre Experiment
Survey
Tic Tac Toe
Training Session
9
METHODOLOGY

17. 10
PROTOCOL
Pre Experiment
Survey
Tic Tac Toe
Training Session
Review
Profile Page +
Pull Request
METHODOLOGY

18. 10
PROTOCOL
Pre Experiment
Survey
Tic Tac Toe
Training Session
Review
Profile Page +
Pull Request
METHODOLOGY

19. 10
PROTOCOL
Pre Experiment
Survey
Tic Tac Toe
Training Session
Review
Profile Page +
Pull Request
METHODOLOGY

20. 11
PROTOCOL
Pre Experiment
Survey
Tic Tac Toe
Training Session
Review
Profile Page +
Pull Request
11
Reasonable Code Snippet:
Unreasonable Code Snippet:
METHODOLOGY

21. 12
PROTOCOL
Pre Experiment
Survey
Tic Tac Toe
Training Session
Post Experiment
Survey
Review
Profile Page +
Pull Request
What elements of the displayed profile or pull request did
you consider when making your decision? *
What strategies do you use to decide to share this content?
Does it vary across each community? *
METHODOLOGY

22. 13
PARTICIPANTS
METHODOLOGY

23. Recruited 42 Participants
13
PARTICIPANTS
METHODOLOGY

24. Recruited 42 Participants Advanced Topics Computer Science Course
13
PARTICIPANTS
METHODOLOGY

25. All familiar with submitting and reviewing pull requests
Recruited 42 Participants Advanced Topics Computer Science Course
13
PARTICIPANTS
METHODOLOGY

26. All familiar with submitting and reviewing pull requests
Recruited 42 Participants Advanced Topics Computer Science Course
13
PARTICIPANTS
METHODOLOGY
Gender Quantity Age Range Country of Origin Minority Not Minority
Men 30 22-33
24 India, 1 Nepal,
1 USA
2 23
Women 12 23-29
2 China, 5 India,
1 Iran, 3 USA
1 10

27. How do programmers review pull
requests?
Where do programmers think they
look vs. where they really look?
What strategies do people use to
manage signals for their personal
identity?
14
RQ2
RQ1
RQ3
ANALYSIS

28. How do programmers review pull
requests?
Where do programmers think they
look vs. where they really look?
What strategies do people use to
manage signals for their personal
identity?
14
RQ2
RQ1
RQ3
ANALYSIS
Fixation Duration
# Of Fixations
Reported Experience

29. How do programmers review pull
requests?
Where do programmers think they
look vs. where they really look?
What strategies do people use to
manage signals for their personal
identity?
14
RQ2
RQ1
RQ3
ANALYSIS
Fixation Duration
# Of Fixations
Reported Experience
Mapped Responses

30. How do programmers review pull
requests?
Where do programmers think they
look vs. where they really look?
What strategies do people use to
manage signals for their personal
identity?
14
RQ2
RQ1
RQ3
ANALYSIS
Fixation Duration
# Of Fixations
Reported Experience
Mapped Responses
Platform Specific
Strategies

31. Social Signals
Code Signals
Technical Signals

32. (AI)
(DN)
(FF)
(RE)
(HM)
(CA)
(RS)
(BC) (AC)
(UD)
(PT)
(SD)
(RP)
(CD)
(TM)
Social Signals
Code Signals
Technical Signals

33. RESULTS

34. TABLE II: Participant Fixations on Areas of Interest
Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
T
= true accept
F
= false accept
F
= false reject
T
= true reject
—
= no decision

35. TABLE II: Participant Fixations on Areas of Interest
Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
T
= true accept
F
= false accept
F
= false reject
T
= true reject
—
= no decision
12 correct decisions

36. TABLE II: Participant Fixations on Areas of Interest
Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
T
= true accept
F
= false accept
F
= false reject
T
= true reject
—
= no decision
12 correct decisions

37. TABLE II: Participant Fixations on Areas of Interest
Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
T
= true accept
F
= false accept
F
= false reject
T
= true reject
—
= no decision

38. Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
Specifically, participants mentioned the correctness of the pull
request, code complexity, and beautification such as style and
Next, we consider the elements our participants reported
considering compared to what they fixated on during the
T
= true accept
F
= false accept
F
= false reject
T
= true reject

39. Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
Specifically, participants mentioned the correctness of the pull
request, code complexity, and beautification such as style and
Next, we consider the elements our participants reported
considering compared to what they fixated on during the
T
= true accept
F
= false accept
F
= false reject
T
= true reject
Participants spent a majority of their time fixating on code
( x̅ = 57.15%, x̃ = 64.23%)
Code Signals

40. Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
Specifically, participants mentioned the correctness of the pull
request, code complexity, and beautification such as style and
Next, we consider the elements our participants reported
considering compared to what they fixated on during the
T
= true accept
F
= false accept
F
= false reject
T
= true reject
Participants spent a majority of their time fixating on code
( x̅ = 57.15%, x̃ = 64.23%)
Participants spent a considerable amount of time on tech signals
( x̅ = 32.42%, x̃ = 28.45%)
Code Signals
Technical Signals

41. Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
Specifically, participants mentioned the correctness of the pull
request, code complexity, and beautification such as style and
Next, we consider the elements our participants reported
considering compared to what they fixated on during the
T
= true accept
F
= false accept
F
= false reject
T
= true reject
Participants spent a majority of their time fixating on code
( x̅ = 57.15%, x̃ = 64.23%)
Participants spent a considerable amount of time on tech signals
( x̅ = 32.42%, x̃ = 28.45%)
Code Signals
Technical Signals
Participants spent a considerable amount of time on social signals
( x̅ = 10.43%, x̃ = 7.38%)
Social Signals

42. Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
Specifically, participants mentioned the correctness of the pull
request, code complexity, and beautification such as style and
Next, we consider the elements our participants reported
considering compared to what they fixated on during the
T
= true accept
F
= false accept
F
= false reject
T
= true reject
5 out of 20 participants spent 48% to 62% of their time
fixating on technical signals and an above average time
on social signals (17% to 31%)
Participants spent a majority of their time fixating on code
( x̅ = 57.15%, x̃ = 64.23%)
Participants spent a considerable amount of time on tech signals
( x̅ = 32.42%, x̃ = 28.45%)
Code Signals
Technical Signals
Participants spent a considerable amount of time on social signals
( x̅ = 10.43%, x̃ = 7.38%)
Social Signals

43. Participants
M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
Experience H H H L H H H H H H L L H L H L L L H L
PR Reviewed P P P A A A A T T T P P P A A A A T T T
Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗
Overview
Code Signals
67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27%
Technical Signals
26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56%
Social Signals
7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17%
Code Signals
After Code Snippet
(AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54%
Before Code Snippet
(BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46%
Technical Signals
Contribution Activity
(CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11%
Commit Details
(CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3%
Contribution Heat Map
(HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18%
Pull Request Title
(PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5%
Popular Repositories
(RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45%
Submission Details
(SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18%
Social Signals
Avatar Image
(AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46%
Display Name
(DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11%
Followers/Following
(FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – –
Repository Popularity
(RE) – – – – – – – 5% – – – – – – – – – 1% – –
Repository Stars
(RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13%
To Merge
(TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21%
User Details
(UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9%
Specifically, participants mentioned the correctness of the pull
request, code complexity, and beautification such as style and
Next, we consider the elements our participants reported
considering compared to what they fixated on during the
T
= true accept
F
= false accept
F
= false reject
T
= true reject
—
= no decision
More details in paper: http://bit.ly/BeyondCode

44. FIXATION + EXPERIENCE = RESULTING GROUPS
RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20
M1, M7, M5, M8
M4,
W2, W4, W6, W8,
W5, W9
M2, M3, M6, M9,
M10,
W1, W7, W10
High-Experienced
Thinkers
High-Experienced
Glancers
Low-Experienced
Foragers
Low-Experienced
Thinkers

45. FIXATION + EXPERIENCE = RESULTING GROUPS
RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20
M1, M7, M5, M8
M4,
W2, W4, W6, W8,
W5, W9
M2, M3, M6, M9,
M10,
W1, W7, W10
High-Experienced
Thinkers
High-Experienced
Glancers
Low-Experienced
Foragers
Low-Experienced
Thinkers

46. FIXATION + EXPERIENCE = RESULTING GROUPS
RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20
M1, M7, M5, M8
M4,
W2, W4, W6, W8,
W5, W9
M2, M3, M6, M9,
M10,
W1, W7, W10
High-Experienced
Thinkers
High-Experienced
Glancers
Low-Experienced
Foragers
Low-Experienced
Thinkers

47. RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK?
PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED
21
Post Survey
Eye-Tracker
Study Phase
0 5 10 15 20
No. of Participants w/ Overlap Between Phases
AC
AC- BC
CA
CD
CD- HM
CDHM- PT
D- RE
D- SD
E- AI
E- DN
FF
RP
RS
TM
UD
AreasofInterest
Post Survey
Eye-Tracker
Study Phase
0 5 10 15 20
No. of Participants w/ Overlap Between Phases
AC
AC- BC
CA
CD
CD- HM
CDHM- PT
D- RE
D- SD
E- AI
E- DN
FF
RP
RS
TM
UD
AreasofInterest

48. 73% reported using the code
snippet to make a decision
RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK?
PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED
21
Post Survey
Eye-Tracker
Study Phase
0 5 10 15 20
No. of Participants w/ Overlap Between Phases
AC
AC- BC
CA
CD
CD- HM
CDHM- PT
D- RE
D- SD
E- AI
E- DN
FF
RP
RS
TM
UD
AreasofInterest
Post Survey
Eye-Tracker
Study Phase
0 5 10 15 20
No. of Participants w/ Overlap Between Phases
AC
AC- BC
CA
CD
CD- HM
CDHM- PT
D- RE
D- SD
E- AI
E- DN
FF
RP
RS
TM
UD
AreasofInterest

49. 45% reported using information
related to previous contributions
73% reported using the code
snippet to make a decision
RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK?
PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED
21
Post Survey
Eye-Tracker
Study Phase
0 5 10 15 20
No. of Participants w/ Overlap Between Phases
AC
AC- BC
CA
CD
CD- HM
CDHM- PT
D- RE
D- SD
E- AI
E- DN
FF
RP
RS
TM
UD
AreasofInterest
Post Survey
Eye-Tracker
Study Phase
0 5 10 15 20
No. of Participants w/ Overlap Between Phases
AC
AC- BC
CA
CD
CD- HM
CDHM- PT
D- RE
D- SD
E- AI
E- DN
FF
RP
RS
TM
UD
AreasofInterest

50. 45% reported using information
related to previous contributions
73% reported using the code
snippet to make a decision
RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK?
PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED
21
Post Survey
Eye-Tracker
Study Phase
0 5 10 15 20
No. of Participants w/ Overlap Between Phases
AC
AC- BC
CA
CD
CD- HM
CDHM- PT
D- RE
D- SD
E- AI
E- DN
FF
RP
RS
TM
UD
AreasofInterest
Post Survey
Eye-Tracker
Study Phase
0 5 10 15 20
No. of Participants w/ Overlap Between Phases
AC
AC- BC
CA
CD
CD- HM
CDHM- PT
D- RE
D- SD
E- AI
E- DN
FF
RP
RS
TM
UD
AreasofInterest
1 participant reported using the
submitter’s profile image

51. RQ3: WHAT STRATEGIES DO PEOPLE USE TO MANAGE SIGNALS FOR THEIR
PERSONAL IDENTITY?
PLATFORM STRATEGIES VARY TO PROTECT THEIR
IDENTITY & CONTRIBUTIONS
22
Nameless code should
stand alone
Full Profile —>Trustworthy

52. RQ3: WHAT STRATEGIES DO PEOPLE USE TO MANAGE SIGNALS FOR THEIR
PERSONAL IDENTITY?
PLATFORM STRATEGIES VARY TO PROTECT THEIR
IDENTITY & CONTRIBUTIONS
22
Nameless code should
stand alone
Full Profile —>Trustworthy
“I’ve different accounts for
the different kinds of work. [...]
It maybe because my code has
nothing to do with my name or
my image, the code needs to
talk for itself.” (S26)

53. RQ3: WHAT STRATEGIES DO PEOPLE USE TO MANAGE SIGNALS FOR THEIR
PERSONAL IDENTITY?
PLATFORM STRATEGIES VARY TO PROTECT THEIR
IDENTITY & CONTRIBUTIONS
22
Nameless code should
stand alone
Full Profile —>Trustworthy
“I’ve different accounts for
the different kinds of work. [...]
It maybe because my code has
nothing to do with my name or
my image, the code needs to
talk for itself.” (S26)
“such familiarity gives a
feeling of trust.” (S15)

54. DISCUSSION + IMPLICATIONS 23
All participants reviewed at least one social signal.
Does viewing content necessarily mean that the information
seen will influence the decision about a pull request?

55. DISCUSSION + IMPLICATIONS 23
GitHub profiles make social + human capital explicit.
How does giving users control over their identity influence
participation?
All participants reviewed at least one social signal.
Does viewing content necessarily mean that the information
seen will influence the decision about a pull request?

56. Summary
Paper: http://bit.ly/BeyondCode
DenaeFordRobin
(BC) (AC)
(UD)
(PT)
(SD)
(RP)
(CD)
(TM)
dford3@ncsu.edu
We observe that both social and technical aspects are being taken
into consideration when deciding upon pull request acceptance.
Moreover, many more social aspects are being considered during
the experiment than reported during the post-experiment survey
Future work will study how the execution of concealing or
amplifying these signals affect developers across the identity
spectrum and development experiences at scale.
Artifact: http://bit.ly/BeyondCodeArtifacts