The Consumer Systems Technology team at Cox Media Group operates more than 80 digital properties, including Atlanta favorites AJC.com, WSBTV.com, and WSBRadio.com. We’ll tell you why and how we use New Relic to monitor and tune the performance of our content management system and web API, and how we plan to use it in the future to make our web sites even better.
You might not know the name Cox Media Group, but you probably recognize the names of our local Atlanta brands. We operate radio stations, TV stations, and newspapers in about 20 markets, and we have more than 80 digital properties.
It’s natural to think about New Relic in terms of Performance. After all, the name of the game is Application Performance Management. But for me, as an architect and product manager, New Relic is about something else.
New Relic is about Scale. In the media business, and especially online news, scale is absolutely key to our success. It only takes one news event, a shooting or a tornado or a snowpocalypse, to change the usage profile of our web sites dramatically. When I say dramatically, I’m talking about an order of magnitude increase in traffic in a matter of minutes.
When that kind of change in traffic hits your site, it can very quickly reveal weaknesses in your application that you might otherwise never have noticed. These events can overwhelm the available compute capacity at any layer of the application stack and become basically an accidental DDoS.
Let me tell you about the first and biggest time when New Relic saved my bacon. As we were rolling site after site onto our new CMS platform, we began to see some serious scalability issues. Pages were loading far too slowly, updates were delayed, some components failed to load at all, and the user experience was generally getting much worse. Operations reported the servers were massively overloaded and throwing huge numbers of errors.
For large applications like our CMS, this is the most valuable screen in New Relic, especially when you are hitting limitations of scale. This screen tells you exactly where your application is spending its compute time, therefore what you need to change to take pressure off of a struggling compute cluster.
What were the servers working so hard to do, exactly? The number one thing was something very odd. It was building the little weather widget that lives in the upper corner of our pages. In fact it was building that widget way more often that it was building the pages themselves!
It turned out, the top 3 requests were eating more than 2/3 of the system’s capacity, and all 3 of them were AJAX requests. The pages themselves were being served from cache, you see, but the AJAX requests had been marked uncacheable for various accidental or short-sighted reasons. That was absolutely fine under normal traffic, but when a traffic event hit, it caused the load on our server to increase super-linearly.
Needless to say, we tackled these problems in a hurry, and thanks to New Relic, the system became rock solid with just a handful of user stories completed. When a serious weather event quintupled traffic to our Atlanta properties a few weeks later, the servers did not even break a sweat.
Today we use New Relic as a regular part of our operations. For the CMS we report performance numbers to stakeholders on a daily basis. These KPIs include the server response time and the page load time.
Monitoring
For our new Content Search API, which powers several tools internally as well as mobile apps, New Relic is even more pivotal, because we use it not just to report KPIs but also for production monitoring. New Relic’s alerts are sent to PagerDuty when the system becomes unavailable or performance drops past our reporting threshold.
Although there are open source solutions to real user monitoring, rolling our own solution to paint a picture -- no matter how rosey -- hasn't been tenable for us. The pluggability of New Relic RUM with the browser tool gives us much broader and deeper insights to what our end users are actually experiencing and what might be effecting their optimal UX. What we are working on now is an upgrade to our agents to facilitate better real user monitoring of our apps.
One of the big selling points for us with New Relic when we started on the python beta was the promise of better browser instrumentation. The browser tools provide a wealth of insight into how well the client side is performing. This is crucial to having a well oiled machine since 80% or more of your website load is related to the front end. Once we have our python agent upgraded for our current platform we are looking forward to having more granularity in the monitoring of our front end apps. That granularity of the client side hooked with our own instrumentations will help to reduce our life cycle overhead.
We have a core CMS with millions of users that we currently have New Relic monitoring. We are constantly iterating our core products and finding better ways to monitor their performance. Along with our homegrown feature flags and A/B tests, we use New Relic to give us the real time feedback we need once our products are in production. In the future, we intend to further integrate New Relic into our Wordpress blogs, mobile, and other apps.
Provide better transparency of our data analytics to our customers. Facilitate dialogue of how well products are performing over time and where we should focus our efforts.
At Cox Media Group we find New Relic to be an inherently valuable tool set, and we continue to look for new ways to leverage our relationship to improve operations and deliver real value. Now we’ll be happy to answer any questions you may have.