3. TRACKING PERFORMANCE METRICS
Back End
Labels
What constitutes success
Front End
Tests
Selecting content
Teasing
Time (of posting, spacing)
5. BREAKDOWN OF CLICKS AND SHARES
0 to 2 clicks
77%
3 clicks
(average)
10%
4 to 9 clicks
10%
10 to 30 clicks
2%
40+ clicks
1%
CLICKS
0 shares
80%
1 share
13%
2 shares
5%
3 to 9 shares
2% 10+ shares (3)
0%
SHARES
6. STANDOUTS: TOP PERFORMING CONTENT (40+ CLICKS)
1. KKK, new Black Panther Party plan simultaneous rallies in South Carolina – 77 clicks, 18 shares
2. ‘Extremely venomous’ snake report has Georgia rangers on high alert – 61 clicks, 4 shares
2. Bridal collection seized by Feds hits the auction block in Atlanta – 61 clicks, 4 shares
3. End of fun: Bubble wrap will no longer ‘pop’ – 53 clicks, 14 shares
4. Nurse who dropped newborn won’t lose her job – 49 clicks, 7 shares
Tease: The hospital’s coordinator of community relations said “The nurse will not be terminated, but we will follow internal
protocols on what type of disciplinary action she will face
5. Police: Body in Florida Walmart goes unnoticed for days– 43 clicks, 2 shares
6. Sister of accused Charleston shooter asks for donations – 43 clicks, 2 shares
7. New state law allows officers to break into hot cars to rescue pets – 42 clicks, 13 shares
Tease: Violators may be fined up to $125.
7. TEST 1: TEASED CONTENT
A- Tease less: Highest average, highest number of
clicks, lowest effort
B- Pull Quote: Slightly below average, possibly
due to small sample size, shows potential.
C- Quick Fact: Average, works when relevant
D- Subjective: Poorest performing, most risky
0 5 10 15 20
Subjective
Quick Fact
Pull Quote
Teaseless
Clicks
Average >3 clicks (%) >10 clicks (%)
8. TEST 2: LOCATION OF CONTENT (ONLY ADDED IF
APPLICABLE)
A- Georgia: added if posts centers on Georgia
subject, top performing,
B- US: added if a state or other city is mentioned
in the headline or tease, doesn’t seem to
discourage audience
C- International: added if a country is mentioned
in headline, not suggested unless it is a major
news event, like Iran deal
0 5 10 15 20
International
US
Georgia
Clicks
Average >3 clicks (%) >10 clicks (%)
9. TEST 3: SOURCE OF CONTENT
A- Audience: Shows potential
Morning/Afternoon digests: Average 3 clicks
B- AJC/Blog: Doesn’t perform unless it pertains
to trends (i.e. Bobbi Kristina) despite Atlanta
audience
C- CMG/National: Does well with optimized
headlines
Subjective click average: 2
Non-subjective click average: 3
5/26 – 6/26 click average: 2
6/27- 8/6 click average: 4
D- NDN: Lowest results, only topics that would
have done well anyway performed 0 2 4 6 8 10 12 14 16 18
NDN
CMG/National
AJC/Blog
Audience
Clicks
Average >3 clicks (%) >10 clicks (%)
10. TEST 4: CATEGORY OF CONTENT/ HOT TOPICS
A- News
Technology/ Science/Weather: 2 click average
Crime: 5 click average
Health: 3 click average
B- Sports: 3 click average
Georgia sports: 3 click average
C- General/Entertainment
Animals: 6 click average
Snakes, Sharks, other scary creatures: 8 click average
Food: 4 click average
Celebrity: 3 click average
11. TEST 5: SPECIAL TOPICS CONTENT
A- Charleston/ Confederate Flag
Average: 6 clicks
Big clicks came from news about rallies, sister asking about donations – both things audience can participate in
B- SCOTUS Gay marriage decision
Average: 2 clicks
Big clicks came from where to get married, where to celebrate in your area
C- Election coverage
Average: 2 clicks
Big clicks came from stories about candidates doing non-political things, who has the most money
14. THINGS TO NOTE
Our success is relative
We reached 10K followers on July 15. Money backing ads decreased after that.
We changed our tone July 1 from witty/sarcastic to straightforward and began experimenting with teaseless.
Number of likes matter
Total average: 3 clicks
5/26-6/26 average: 2 clicks
6/27-8/6 average: 4 clicks
Correlation ≠ Causation
Since we don’t post duplicates, we are not able to tell if a post did well because it was tease less or because it was about
the confederate flag or etc.
Human error exists
Some topics were difficult to monitor because they cross the border and transcend labels.
Caitlyn Jenner (celebrity) spoke at ESPYs (sports) about transgender rights (news).
There were two different people monitoring the project and our goals transformed throughout the process
15. AUDIENCE
Almost 70 percent of fans are female.
13 percent are from Atlanta. Another 8 percent
are from cities surrounding Atlanta, such as
Decatur and Marietta. 35 other Georgia cities
were represented.
Almost 3 percent don’t speak English.
Likes easily digestible news, but don’t limit them
to that.
Likes actionable news.
Keep experimenting!
Atlanta
22%
Augusta
7%
Columbus
6%
Macon
6%
Savannah
5%Rome
4%
Albany
4%
Valdosta
3%
Athens
2%
LaGrange
2%
Other Georgia
40%
Outside US
0%
LIKES
We approached this project from two different perspectives: what we can do before publishing (selecting content, changing teases, posting at peak hours, spreading posts out throughout the day etc.) and what can do once its published (sorting through labels, deciding what is considered successful.)
The first approach was to define thresholds. Most posts, about 77 percent, got between 0 and 2 clicks. Almost 10 percent got the average of 3 clicks. So that meant a good post got between 4 and 9 clicks, which was about 10 percent of the time. A really good post got between 10 and 30 clicks (almost 3 percent). And an outstanding post got more than 40 clicks (less than 1 percent). This system, based on the number of clicks each post received, let us decide what we consider success and failure. We thought we’d look through each group and figure out what trends each share, instead of worrying about labeling.
One interesting thing to note is just because someone doesn’t click on something, doesn’t mean they won’t share it. Almost 3 percent of the time, shares out numbered clicks.
For the rest of our data, we focused on clicks. Because there were so few shares overall, we didn’t see much benefit distinguishing between 1 share and 0 shares.
So here are the posts that were at the top of the list.
As you can see, trying to discern trends from this list is challenging. Trying to do that with more than 400 posts in each group is even more challenging.
So this method didn’t work for three reasons: 1- the same category of content was found in low performing groups and high performing groups 2- it doesn’t give much explanation as to why or what we can learn and change from this. And 3- while there are defined thresholds, the difference between say 39 clicks and 40 clicks doesn’t seem significant to consider a different trend
So from there, we decided to try a labeling system and institute tests of what we were trying to measure.
Our first experiment was teased content.
When we first began SnapNews, we wanted to experiment with tone. We hypothesized that an audience seeking alternative news would be more open to commentary in teases. Everything from funny or sarcastic (Way to stick it to him, porcupine) to commands (Be careful out there) to condolences (R.I.P.) to personal opinions (use of I or We) to leading suggestions (How cool would this be?) to generic sentiments (Aww, Yum) was housed under the category of Subjective. Though we stopped posting subjective posts near the end of June when we only had about 3,000 likes, subjective never gained considerable traction. Only 5 percent received more than 3 clicks and no post was able to get more than 2 shares. Making posts subjective is risky already and doesn’t seem to provide any return on investment. We don’t recommend any more subjective posts. Though quick fact had a higher average number of clicks than pull quote, we suspect a smaller sample size of pull quote posts may be influencing the data. 15 percent of quick fact posts received more than 3 clicks and 16 percent of pull quote posts received more than 3 clicks. However, the quick fact posts that did perform received more clicks than any of the pull quote posts. All this is to say, the real winner here is teaseless posts. With 18 percent of posts receiving more than 3 clicks, we see the most success in this area. Even though most of the teaseless posts were made beginning in July, with more of an audience, we see a consistent trend. By averages, the success of teaseless posts seems negligeable. It’s only one more click per post than other forms of teases. But considering making posts teaseless requires no extra effort, we recommend teaseless content, unless you’re sure a tease is worth it. So making posts with quick facts or pull quotes doesn’t hurt, but neither does teaseless.
Teaseless: 367, Pull Quote: 75, Quick Fact: 181, Subjective: 273
The location of the content was only added as a label if applicable. It usually meant that a city, state or country was mentioned in a headline. If it was a general news, entertainment or sports story that wasn’t centered around the location, but had a city, state or country listed in a dateline, it wasn’t labeled for location. However, for Georgia, if it related to Atlanta or Georgia, even without mentioned Atlanta or Georgia, it was tagged. Charleston/South Carolina and Chattanooga were excluded from this labeling method, except for the breaking news posts. 19 percent of posts covering Georgia-centered topics received more than 3 clicks, with an average of 4 clicks per post. Still most of the top performing Georgia content related to trending topics, such as the Confederate Flag/ Stone Mountain debate or Bobbi Kristina. 15 percent of posts with a U.S. city or state in the headline received more than 3 clicks with an average of 4 clicks per post, and every post received at least one click. And international posts received an average of 1 click per post. We hypothesized that Georgia content would be clicked most considering most of our audience lives in Georgia. What this suggests is not that we should be focusing on national content, but rather that putting another city or state in a headline does not scare off or bother our audience. Other countries, however, does seem to.
The next thing we wanted to test was the source of the content.
Audience includes anything that comes from our department, whether it’s PT’d, aggregated or original reporting or Access Atlanta. An AJC or Blog story includes anything that originated from AJC, so an entertainment gallery, a sports update, a political insider story, a Luckovich cartoon or opinion, the Atlanta Now breaking news team – anything AJC that’s not Audience created. No Superstubs were tested. CMG/National is really anything that was sent through email from the National Content Team. It also includes WSB stories. The NDN videos posted on Snap News were the same NDN videos posted on the main AJC Facebook account, with the same headlines and teases. With our knowledge of how NDN videos perform on the main account, we had low expectations for Snap News. We also hypothesized that our audience would be most interested in National Trending news, versus general AJC beat coverage news.
So what we found.
Basically, our hypotheses were correct. NDN posts didn’t do very well. Stories about sharks and snakes did well, but those did well across sources, making investment in NDN unnecessary. Neither did AJC content. We think there is more potential experimentation to be had with AJC Content. But from what we found, the only interest in that category is content pertaining to big news stories, such as Bobbi Kristina. Despite the fact that almost all of our audience is from Georgia, including 13 percent from Atlanta, general state news, local sports nor Georgia politics interest them. The two strong groups are National Content Desk stories and Audience stories, which isn’t surprising considering most of that content is trending news. The top performing audience content was the story about the Federal bridal collection coming to Atlanta, which received 61 clicks. The other top posts (more than 3 clicks) contained a mix of PT, original reporting, digests, lists and galleries. We see the most potential with digests, which received an average of 3 clicks. One big thing we want to point is that almost all of the best performing content overall was CMG Content. Additionally, CMG content was the widest source of content, comprising 78 percent, so we were skeptical that certain factors may have affected its results. We stratified based on publish date and tease, and found that tease less posts when we had a larger audience performed better.
Overall, its all about selection. Source of content doesn’t matter so much as long as the content pertains to hot trending topics, which is a good transition into our next slide…
Our original categories of content we were classifying were News, Sports and General/Entertainment. However, unlike the other tests, assigning labels was largely up to opinion, as posts don’t follow an all or nothing principle, and most posts fit into multiple categories.
So for our purposes, we were more interested in sub-topics within each category as larger content trends that continuously reappear.
The best performing categories were crime, animals and scary creatures. Why? They are all things people are afraid of and need to click on to see how they can avoid. The crime category is anything that relates to police, including positive and negative stories, murders, anyone shooting anything, theft, drugs and alcohol abuse, even crime that is “funny.”
These tags were used in addition to the content type labels to track how well major news events and subsequent coverage performs. We published a huge portion of content surrounding these areas. Our goal for this category was to find out how long our audience retained interest in a certain subject.
The shooting of nine people at a historically black church in Charleston South Carolina was special because it a story that transformed from a breaking news crime story to a conversation about race relations to a discussion of heritage all with a political angle. The first story posted about that area was the June 18 breaking news story. The last story was a Luckovich political cartoon about heritage.
This graph shows an encouraging trend. We have maintained our audience even after ad promotion stopped. However, in order for Snap News to grow, it will need money backing it.
We didn’t reach 10,000 followers until July 15, with money backing our numbers. After that, money to the account was reduced and our audience grew much more slowly. Because we hovered around very low audience numbers for a while, our lack of clicks can partially be attributed to lack of likes. However, five of the top seven performing posts were published before we reached 10K likes showing that if content is really good, people will click. Additionally, we found that our audience did not respond to subjective teases as hypothesized. Posts were mostly subjective until July 1, so couple with a low audience, those factors contribute to our numbers.
We also have no way to prove what causes content to perform. We don’t know if the KKK and Black Panther Party rally story in South Carolina did well because it was about the Confederate flag debate or because it was tease less, or etc.
Finally, because two people were controlling the project, and we each have different perspectives, our numbers were affected. They are 99 percent accurate, but we are leaving a 1 percent room for error due to that fact. Also, some topics transcended borders. Should Caitlyn Jenner, who is a celebrity, speaking at the ESPYs about transgender rights be considered an entertainment, news or sports story?
A breakdown of who we are serving -- 69 percent of our audience is female and 63 percent are from Georgia. The pie chart shows our Georgia audience. While the largest city represented is Atlanta, 79 percent are from outside of the metro Atlanta area.
So, does the Snap News audience like Snap News? Yes, but it’s not the only thing they like. None of the top seven performing posts contained lists.
What this audience searches for is news that allows them to take action – stop doing something, donate, change eating habits, find the best place to go on the weekend, etc.
Overall, Snap News has potential.