Corporate web sites getting hacked is news.  Corporate news sites getting hacked is news.  News sites getting not hacked but going down anyway is…?   When NYTIMES.com went down this week for a couple of hours, they felt they needed to provide the proper context for their downtime.   The headline of the article they published read “Times Says Web Site Failure Is Not a Result of Cyberattack”.  This was not a “note to our readers”, this was presented as a news article.

(http://www.nytimes.com/2013/08/15/business/media/new-york-times-web-site-returns-after-hours-offline.html)

Denials are sometimes newsworthy after all.  “I never took performance enhancing drugs”; “I did not have sexual relations with that woman” and the granddaddy of them all “I am not a crook”.

This past week, just by coincidence it would appear, was the week of news sites having problems.   Less than 24 hours after NYTIMES.com experienced its outage, TIME, CNN and The Washington Post had their ad network, Outbrain, hacked via social engineering.

The two events  and how they were announced serve as an interesting contrast in publicizing “breaches”.   I’m using the term breach loosely here because the NYTIMES downtime certainly may not have been the result of a hack.

So first let’s look at Outbrain.  We know this was a hack because the Syrian Electronic Army (SEA) claimed responsibility for it very publically.  The access was through a phishing attack in which employees received an e-mail from [not-really-the] CEO asking for their logon credentials.  Someone who was asleep when the security folks mentioned “never provide your user ID and password to ANYONE if they ask for it in an e-mail” seems to have been taken in.

The folks at Outbrain Outdid themselves in Outing the breach.   They schooled all others in how it should be done.  They published a chronology of what happened and their response—in detail.

Here is the timeline of events on August, 15, 2013: (all times are EST)

8:40am  SEA began making configuration changes

10:23am  SEA took responsibility for hack of a specified news organization, changing a setting through Outbrain’s admin console to label Outbrain recommendations as “Hacked by SEA.”

10:34am Outbrain internal staff became aware of the breach

10:40am Outbrain network operations began investigating and decided to shut down all serving systems and block all external access

11:03am All systems were shut down

11:50am First communication sent to our clients alerting them that service is suspended

 http://www.outbrain.com/blog/2013/08/additional-details-on-outbrain-phishing-attack.html

This may not be the first time a company came this clean, but it is the first time I’ve ever seen it.   Compare this to the announcement that Twitter made when they were hacked.  I write about that at http://wp.me/p2Ob8g-2s

And this is remarkably  different than the approach the Times took.  Although, interestingly enough, the “paper” decided that one bit of chronological detail would be worth providing.

According to a spokeswoman for the Times, “The outage occurred within seconds of a scheduled maintenance update being pushed out, and we believe this was the cause”.    They are able to say the outage occurred “within seconds” of an event that they report happened “about 11:10 a.m.”   While it is difficult to understand how they can be sure that something happened within seconds of an event they cannot pinpoint the time for, that is not the most noteworthy thing about this.  The article tells us that the “failure took place during the peak hours for traffic to the site, between 10 a.m. and 4 p.m.”.

Given 24 hours in the day, if you have a window of 6 of those hours when you know  you experience your “peak traffic”, any IT operations person will tell you that you do NOT schedule maintenance updates one hour into that window.    And usually if there’s one thing New Yorkers know how to avoid, it’s heavy traffic.  (Digression alert: I grew up with my uncles arguing with phrases like “No you don’t take the FDR, you’ll get murdered, go up the West Side and cut over to the Deegan when you get through Harlem” to which my other uncle would respond “You’re nuts, the Yankees are playing, the Deegan will be a parking lot; stay on the West Side and take the Saw Mill to the Cross County or cut over at Mosholu”)  But not in this case.   If we follow the “traffic” metaphor, someone decided to do road work at rush hour.

In addition, routine maintenance should not be performed unless it is tested, right?  NIST, in describing promoting patches and other configuration changes, provides this warning:

Patches and configuration modifications should be tested on non-production systems since remediation can easily produce unintended consequences.  Many patches are extremely complicated and can affect many portions of a system, since they often replace system files and alter security settings.  Patches may also include fixes for multiple vulnerabilities or contain non-security changes, such as new functionality. In addition, patches and configuration changes are often released in haste to repair a vulnerability quickly, and therefore often receive less testing than the original software. Installing patches, modifying configurations, and uninstalling software may change the system behavior such that it causes other programs to crash or otherwise fail.

http://csrc.nist.gov/publications/nistpubs/800-40-Ver2/SP800-40v2.pdf

So what happened?  Why was routine maintenance scheduled for a time of peak traffic?  Why does it appear that it was not tested sufficiently before being put in production?  And why in the world is this newsworthy?  We don’t actually get answers to these questions, but the rest of the news that’s fit to print goes from glib to grim.

In fact, the rest of the article displays the uneasy transition that is taking place from print journalism to the on-line world and then follows it up with some innuendo.

We are told that the Times used Facebook to publish breaking stories on the violence in Egypt and that the clever folks in the editorial department offered to tweet Op-eds in “140-character increments”.

The glib section ends when the article draws a comparison between having your on-line news delivery site, NYTIMES.com, down and the “East Coast Blackout” of 2003.  Millions of people without power?  Tens of thousands of emergency personnel mobilized across some of the most densely populated areas of the country.  The Times tells us that “the failure was reminiscent of a power blackout”.  Maybe at the offices of the Times it was like a power failure, but for most of us, it just meant clicking on the next news site  in our “favorites”.

Now comes the grim, kind of creepy part of the article.  Having compared themselves to a utility providing power to the East Coast, the Times now reports that when they went down “Others were there to fill the void”.  In fact, it appears that The Wall Street Journal provided “Bonus lunchtime reading: WSJ.com is free to everyone for two hours”, roughly the same time as the outage.   The Times is quick to tell us that its competitor has done this in the past when there was breaking news.  More denial, at least implicitly: them offering free coverage when we’re down are not related to each other.

There is clearly a hint that the New York Times Corporation, while not wanting a lawsuit on its hands and not pointing any fingers directly and while specifically stating this is not, not, not a cyberattack, appears to want us all to know that IF, and it’s a big “IF”, someone wanted to take advantage of the situation, their competitors could/would/(did?).   No one is accusing anyone of industrial sabotage here, you understand.  Nothing to see folks.  Move along.  In fact, far be it from us, here at the Times, to accuse anyone of sabotage (except the Chinese in January did try to hack us, don’t forget).  However, we here at the Time would just like to conclude our article which describes a maintenance outage that went awry by mentioning that “Ezra Klein of The Washington Post posted ‘What, you thought Jeff Bezos was going to buy the Post and play defense?’”   Not that we at the Times are saying that the man who runs Amazon and now owns one of our competitors would know anyone who could bring down our website or anything.

This leads us to a new principle for announcing outages/breaches: it’s better to give detailed facts than to sound paranoid.