Cyber Asset Lesson Learned Finalized

NERC’s Standards Committee (SC) approved the Lesson Learned (LL)document on CIP-002-5: BES Cyber Assets on January 21, 2015. This document was originally released for comment on September 9, 2015, and the final document is dated December 7, 2015. It was created by NERC’s V5 Transition Advisory Group (V5TAG).

One of my criticisms of the LL process is that the LLs will be commented on, get revised, and then get approved and posted by the SC without industry having a chance to see or vet the revisions before they are finalized. While NERC has re-posted some of the LLs for comments, they’ve generally only done that when there are major revisions, and they definitely haven’t done so for all (or even most) of the Lessons Learned documents. This leads to a limited number of people actually having input into what the final, official, document contains. I believe this process has the potential to introduce flaws into the documents, and it happened again with the Cyber Asset LL.

The final document will be available from the “Related Documents” section of the CIP-002-5.1 page at NERC’s website, and the consideration of stakeholder comments will be available from the transition program page. The redlined version of the document is a little harder to find, but it is available from the SC agenda package for their January 21, 2016 meeting.

Sidenote: At one point, the redlined versions were not provided by the V5TAG or the SC. As part of my day  job I actually went and created a redlined version from the previous draft and the finalized draft for a couple of the LLs. That was a pain-in-the-butt, so the inclusion of the redlined versions the last few times finalized LLs were released was much appreciated.

Anyways, the purpose of this post is to look at some of what is in the LL and some of what was taken out of it.

When Cyber Assets meet a threshold of BES impact they become BES Cyber Assets (BCA) which may be grouped by responsible entities into BES Cyber Systems (BCS)

While I don’t disagree with this statement, I think it may be misleading to some people, specifically the word “threshold”. If the Cyber Asset has an adverse impact on the BES within 15 minutes of being misused, degraded, or unavailable, then it is a BES Cyber Asset.[^This is a simplified definition, obviously, and there are more qualifiers and requirements, but this summary works for now.] While the 15 minutes could be considered a threshold, there is no allowance in the standards for the adverse impact to have some minimum threshold. The way the sentence in the LL is worded, with “threshold” directly preceding “of BES impact” could cause some confusion for somebody who associates “BES impact” with the “adverse impact” from the BES Cyber Asset definition.

Some study participants assessed each functional system at a site or facility to determine its potential to adversely impact the BES in 15 minutes or less…Other study participants identified all Cyber Assets, grouped them into BES Cyber Systems, and evaluated the impact of the resulting system.

I like how this provides two approaches to achieving the goal of identifying BES Cyber Assets. This is a good example of an LL being written in such a way that it acknowledges that, “there may be other legitimate ways to fulfill the obligations of the requirements that are not expressed within this supporting document,” as the LL puts it in the introductory paragraph.

Did the device’s function directly impact the reliable operation of a BES asset?

The word “directly” was removed from here, and there were similar edits at other places. This was done in response to a comment from ACES, and seems like a good decision, although it leads to a problem discussed next. There’s nothing in the definition of BES Cyber Asset that talks about a difference between “direct” and “indirect” impacts, and so adding that concept here seems like it would add to confusion insteading of making things more clear.

Did the device function as an EACMS, PACS, or Intermediate System? These types of devices, such as firewalls, intrusion prevention systems or physical access controllers and others that perform security functions may indeed have an adverse impact if they are unavailable or misused to allow unauthorized access or deny authorized electronic or physical access, but it is an indirect impact. These devices have their own definitions and requirements in the CIP version 5 Reliability Standards and therefore are not considered BCAs due solely to their impact.

This is the most glaring example in this document of the problem of not having LLs vetted by industry after being edited. I do not think these changes would have made it through a public vetting process, if there had been one. In the first part of this paragraph, they are saying straight-up that firewalls, IPSs, or other security devices can have an adverse iimpact if they are unavailable or misused. While these devices do have their own definitions, there is nothing in the definition of BES Cyber Asset that would imply that an entire class of devices can be categorically excluded from the requirement to determine whether they are BES Cyber Assets. By removing those last five words from this paragraph in the draft LL, the V5TAG has gone beyond providing an example of a compliant approach, and has gone into the terrority of interpreting the standards.

There are programmable devices that may have an impact, but have no way to change their executing code, have no concept of a user or authentication, have no ports/services, have no network connectivity, no concept of event logs or alerting, no patches or updates, andor are located in areas where physical security perimeters cannot be established. For these devices, the ERO will allow (for the purposes of assessing compliance with the standards) Responsible Entities to only protect those devices to the extent capable.

The last sentence in this excerpt had, by far, the most comments related to it. EEI went so far as to say that unless the sentence was removed they couldn’t support the document being submitted to the SC for approval. The problem with the sentence was that it implies that non-programmable devices, if they would have an adverse impact within 15 minutes of their misuse, degradation, or unavailability, would have to be protected to the extent capable. The definition of Cyber Asset, though, explicitly states that a Cyber Asset is programmable. That means that a number of new devices would now be in-scope for the CIP standards.

They changed this table to say they were identifying Cyber Assets typically “evaluated” as BES Cyber Assets instead of saying they were typically “identified” as BCA. This change seems to be more in line with what the purpose of the list of device types originally was.

The transition study participants found a lack of clarity in the CIPV5 Reliability Standards because they do not specifically define the “programmable electronic device” component of the BES Cyber Asset NERC Glossary term. Consequently, the CIP V5 Transition Advisory Group referred the identified issues to be evaluated for standards development.

This was added to the document. While a better definitio of Cyber Asset or a definition of “programmable” is needed, unfortunately the creation of an SDT will not be a very quick process, so that clarification likely won’t help most entities very much in the near- to mid-term future.

Another Example of Why Prescriptive Regulations Fail

Last week, the Department of Homeland Security’s ICS-CERT released an advisory for the Advantech WebAccess product. The advisory detailed 15 different vulnerabilities, reported by at least seven different security researchers. They said that the vulnerabilities could be exploited remotely, and that an attacker with “low skill” would be able to eploit the vulnerabilities.

Patrick Coyle pointed out that the release notes and press release for this update, released by Advantech in late December, don’t have any indication that the update has any security-related patches. This, despite having 15 different vulnerabilities being fixed!

For people who are lucky enough to live in the NERC CIP world, this creates an interesting situation. Organizations are required to monitor a “patch source” for cyber security patches. This is likely to be the vendor, in this case Advantech. (Note: I don’t want it to sound like I’m picking on Advantech. The same situation happens with other vendors, too.) If the patch isn’t security-related, then the organization is not required to apply the patch. If the patch is security-related, then they either need to apply the patch or else create a mitigation plan for how they will prevent the vulnerability from being exploited.

Since the Advantech release notes didn’t say anything about this patch being security-related, the organization would have stopped there. Now, a month later, ICS-CERT releases their advisory and we found out, “Oh, that patch was security-related.” Here’s the kicker, though: The organization isn’t required to scour the Internet looking for vulnerability reports. They aren’t required to monitor ICS-CERT, they’re not required to monitor Full Disclosure, nothing. They did what they were required to do when they checked Advantech’s release notes and didn’t find anything that said it was security-related.

So now, we’re in a situation where the organization has a system with at least 15 publicly disclosed vulnerabilities. For the CIP patch management process, they did what they were supposed to do. The compliance people checked to make sure that they evaluated the Advantech patch. The operations people have a product they’re using, and if it’s working, they aren’t going to want to apply an update. The security people may or may not exist, and if they do exist, they may or may not be monitoring ICS-CERT (although I hope they would be).

The bottom line is, you have an entity which is fully compliant with the NERC CIP regulations, which have regulations in place requiring you to apply security patches, yet this security patch hasn’t been applied.

Ted Koppel’s Lights Out is Ridiculous

Ted Koppel released a book, “Lights Out: A Cyberattack, A Nation Unprepared, Surviving the Aftermath,” A couple weeks ago. The book was not received all that well by the electric industry. I haven’t read the book yet, mainly because I don’t want to add money to the sales figures. I’m sure I could find it somewhere online on that Deep, Dark Web, but I generally don’t pirate things just on principle.

He has given several interviews as part of his book tour, though. Among them, one was on the Diane Rehm Show, which has the transcript available here, another was with Hugh Hewitt, with the transcript here, and one with CSO Online available here.

Koppel (from Rehm Show): as great as the tragedy of Paris is, it’s still a conventional form of terrorism that tragically we have grown accustomed to over the years. And what worries me far more is if a group like ISIS, which already has a cyber war capability, and is already probing on the cyber front, they will have the capacity to reach out from wherever they are in the world.

Right at the start, I have to question his terminology. ISIS has not shown any “cyber war” capabilities. This grossly hyperbolizes what ISIS has done. While their propaganda machine uses technology is has been successful, that does not mean that they have a “cyber war capability.”

Koppel (from Rehm Show): There is no way – I mean, the notion that we can keep out terrorists when the terrorists end up being 19 and 20 years old and if you ran into them in the street, you know, unless you can look into their hearts and minds and see what’s going on there, how can you tell?

This is one point where he does make sense. One of the problems with the “War on Terror” idea is that the U.S. tried to fight a war against an idea by using physical troops and weapons. Unless you’re willing to simply exterminate everybody who holds, or likely might hold, the view you disagree with, that isn’t going to work.

Koppel (from Rehm Show): And what ISIS is successfully doing is imbuing in us a sense of fear and suspicion of all Muslims, which is just going to be devastating to the Muslim community, but also devastating to us.

Koppel has talked and been around politics for a long time. When he’s talking about that, he actually seems to be making sense. This is one of the problems with the “Keep all Muslims out of the US” idea.

Koppel (from Rehm Show): We have become now so dependent upon so many different aspects of the internet, that we fail to see that the internet has now become a weapon mass destruction.

And now he starts to go off the rails. To compare the Internet to a WMD is ridiculous.

Koppel (from Rehm Show): They have plans for every possible natural disaster, but the plan is get yourself a two to three supply of food, make sure you have a radio with adequate batteries. Make sure you have flashlights. Make sure you have water and enough medicine to take care of you for two or three days.

I’ve been to presentations by disaster recovery and emergency preparedness professionals. This does not describe what they do. The problem is, the level of follow-through by people in the public is not that great. They give smaller advice like this because they know that if they have this level of advice, people might follow throug on some of it. If you tell them to have a month’s worth of food ready, most people aren’t going to do anything because they don’t know where to start for a problem of that scale.

Koppel (from Rehm Show): the chamber of commerce has done such an effective job of blocking cyber regulation because there are concerns the power industry is deregulated. On one level, that’s good for all of us because it brings the price of electricity down. But that deregulation means it’s not subject to, as the term implies, it’s not subject to very much federal regulation.

This is disingenuous, at best. Koppel’s book is about the Bulk Electric System (BES) going down, not just an outage at some municipal power company or something. And the BES does have cybersecurity regulations, as well as reliability regulations. There may be some debate over how good they are, but it is regulated.

Koppel (from Rehm Show): the deputy director of FEMA is a former vice admiral in the Coast Guard – a very nice man and couldn’t have been more gracious to me – but he said, Ted, I just think you’re wrong. I don’t think this is going to happen. I don’t think we’re vulnerable to this kind of an attack.

…taking out, let’s say, the City of New York? What would you do? Well, he said, I’d have to evacuate.

The next day, I went to see his boss, the administrator of FEMA. Yes, he said, he is absolutely convinced that this can happen and very likely will happen. Well, I said, what happens if the attack targets New York City? Do you evacuate? Oh, no, he said. You can’t evacuate New York City. Too many people. Where are you going to put them? Now, here are the two top people at FEMA in total disagreement, A, about whether it can happen and, B, about what you would do if it does.

This is completely believable (and I’m not being sarcastic). Isn’t bueracracy great?

Rehm: But isn’t NERC the National Electric Regulatory Commission?

Koppel (from Rehm Show): No, NERC is actually the industry body.

I don’t think most people in industry would take this view. The short version is, NERC (the North American Electric Reliability Corporation) is tasked by FERC (Federal Energy Regulatory Commission) with writing and enforcing the rules. Whenever NERC writes the rules, though, they have to be approved by FERC before going into effect.

Koppel (from Rehm Show): But in the final analysis what journalism ought to be about is alerting the public to potential danger.

This is the problem with journalism. Journalism should be about informing and educating. When you view your job as reporting on every threat and every possible danger is when people get an unrealistic view of what dangers they actually face in life. It’s why more people are scared to fly in a plane than ride in a car, even though airplane travel is demonstrably safer.

Koppel (from Rehm Show): There are 3,200 companies out there and several hundred of them are not that big, they’re small. They’re not that wealthy. They don’t have the money to spend on cybersecurity. And if you can get into one of those – and that’s pretty easy – then nations like the Chinese and the Russians have learned how to trace it all the way back into the central SCADA systems.

I’m not even sure what he means by this. Is he saying that if an attacker gets a small rural cooperative distribution company, they can then get easy access to controlling the largest high-voltage lines? Because that’s ridiculous.

Hewitt: I go back to the Durkovich interview. I went and looked her up after I read Lights Out, and I’m sure she’s very competent. She graduated from Duke University in 1994, and she’s done a lot of interesting things. But the military people you talk with who are more my age, 59, or have been out of command for a couple of years versus the youngsters, they’re just much more sober about this, realistic about this.

I don’t think that being older leads to a better understanding of the cybersecurity field. Realistically, I don’t think that matters a bit, other than to say that people who are retired, formerly high-level government workers aren’t necessarily who I’d be asking for a current assessment of how things are.

Koppel (From Hewitt Show): And frankly, I’m not an expert on it today. I am simply, as I have been doing for most of my professional life, reporting what people who know more about a subject than I do tell me.

We’ll come back to this one. Just remember, it’s important to pay attention to who you decide to interview when you realize you don’t know about something and want to write a book on it.

Hewitt: There’s no more chilling anecdote than at the 2011 Black Hat conference, Ted Koppel, where you say someone stood up and gave out the password to all of the SCADA’s.

Koppel: Yup, that’s right.

I’m pretty sure he’s talking about this presentation by Dillon Beresford. This literally makes no sense, though. That Koppel would just agree with something as ridiculous as giving out the “the password to all of the SCADA’s” is almost beyond belief. I know he said he wasn’t an expert, but if you’re going to write a book on it, you need to at least realize that doesn’t make sense.

And then there’s the best interview response of them all.

CSO Online: Did you interview penetration testers who have experience in the electric generation/transmission sector for this book?

Koppel: No, I did not.

He writes an entire book, brags about spending a year-and-a-half on it and how many people he’s interviewed, and he didn’t bother to even ask one person who actually has experience in doing what he says is going to happen? If you want to be educated on hacking the power grid, maybe you should talk to somebody who actually knows how to hack SCADA systems.

Hewitt: I would say on Pages 96–99 is perhaps the most, an account of the most disturbing interview I’ve read in a long time. You’re sitting down, and all honor to Jeh Johnson, the Secretary of Homeland Defense, he’s a public servant and a good man, but when you sit down and you talk to him about the threats to the power grid, I quote here, “Johnson’s answer ran slightly more than 13 minutes, and he never addressed the question. It was,” you concluded a little later, “not an area in which he had any expertise.”

I think that last sentence could just as easily have been applied to Ted Koppel, himself.

How Securing the Power Grid is like Teaching Middle Schoolers

Note: This post is based on a 5-minute “Lightning Talk” I gave at the 2015 TCIPG Sumer School. The Summer School is put on every other year, in the Chicago area, and I could not recommend it more. If you are a cybersecurity person wanting to learn more about the power grid, or an electrical/power engineer wanting to learn more about cybersecurity, then you should make a point of attending the next session. The grant money for “TCIPG” recently ran out earlier this year, but the organization is continuing under the Cyber Resilient Energy Delivery Consortium (CREDC) name.

1. Some people just won’t care what you have to say

And sometimes, the person who doesn’t care to listen is the most important person who should be listening. As a teacher, there were students who didn’t think they needed to be in school. I had an 8th-grader tell me, straight up, that he already knew everything he needed to know. He didn’t pay attention in class, didn’t do any work, and crucially, his parents didn’t think his lack of effort in school (failing every class, including PE) was a problem. The dad was a dropout who worked as a logger, and since “his life was fine,” his kid’s education wasn’t important. Despite a number of caring, hard-working teachers at that school, there could be no progress until the child’s (or possibly the parents’) attitude changed.

Similarly, securing our power grid is important. But, unless there is buy-in from corporate leadership, any efforts that a security professional does will likely be insufficient and “band-aid” style fixing of problems.

2. Some people care too much

In teaching, these people are called “Helicopter Parents.” They are the ones who do their kid’s homework, believe that their child can do no wrong, and think that it’s their job as a parent to make sure their child is successful and doesn’t encounter any difficulties in life. I had one student whose mom was on the school board. He did very little work, and when the first report cards came out, he was failing the class. Mom didn’t like that. One teacher, who had taught at the school for over 30 years, said to just change the grade to a “D,” because it wasn’t worth the hassle to deal with the mother. He said he had done that several years prior with the student’s older brother, who Mom had also tried to “protect” in the same way. Another teacher told me that the same older brother was currently in jail on serious charges, after never having to deal with the consequences of his actions while growing up. By caring so much about short-term effects of something like a failing grade, the mom had set him up for failure in the long-term.

When it comes to the power grid, I think the people who care too much are those who think we can protect the electric system from 100% of the threats, 100% of the time. That’s just not a realistic goal, and ignores the “Best Practices” of risk management planning. A similar problem case are those who think that their issue is the only important issue. For example, Senator xxxxxx from Texas had his “pet issue” of electromagnetic pulse weapons. By wearing blinders and expecting industry to follow his ideas, it leads to things like him saying that the electric industry’s actions in the area of EMP weapons should be considered treason. That kind of hyperbole actually hurts your cause, because it leads to industry discounting any any legitimate concerns you have or points you do make.

3. Some people are doing really cool stuff with technology

Technology can make school more engaging for students. I’ve known students who the main reason they enjoyed school was because they were able to be a part of a robotics club. And there are teachers who use technology in their classrooms every day to make school more engaging for students. Likewise, there are some cool projects and tools which can be used by industry to help make organizations more secure. A lot of them, like OpenNSM, are even open source projects which are developed by volunteers, and can be implemented at little to no cost.

4. Money Helps

There’s a reason thousands of teachers are using things like [find name of the crowd source site I used to use] to raise money for their classrooms. It requires money to provide the art supplies which can inspire some kids to be better students, or to provide experiences which make learning more fun and hands-on, like the awesome Outdoor School program in Oregon. In the same way, some security projects just can’t get done without spending some money. For example, the Cyber Threat Alliance is a great project being done by some large security vendors. It requires the vendors to be willing not only to question their business model, but also to be willing to give their employees the leeway to spend time on a project with no direct impact on their bottom line.

5. But, Money Isn’t Everything.

There are some schools doing great things with almost no money. They exist in the inner city and the rural world. And they are proof that things can be accomplished with hard work and commitment, even when the funds might not be there. Likewise, it doesn’t necessarily take money to improve a security program. Maybe somebody will give up an hour of lunch to give a short talk to employees about how to be safe online, or how to make their home wifi network secure. There are little things that can be done, which cost little or no money, which can have a large, cumulative effect on security.

6. It Takes Teamwork

Every great teacher I’ve known has given credit to other people. Whether it’s a principal who creates a great culture at a school, aides who have the patience and skills to work with the hardest students and assist the teacher, previous teachers who have helped cultivate a love of learning in the students, or parents who provide a home environment conducive to learning, there is always someone who has helped to make it possible to connect and have a productive relationship with students.

A security department, working by itself, will not be able to ensure an organization is adequately protected. They will need to work with executives and normal users, work with their vendors, and, yes, even work with their regulators in order to be as successful as possible.

7. Most Important: Dedication is What Leads to Success

There’s a tradition at the Cedar Ridge Outdoor School that (some) leaders will lick banana slugs. I did it when my group was doing a post-activity question/quiz time, an a game called “Stump the Leader.” I said that if my group could beat the leaders at the game, then I would lick a slug. After one of the leaders threw the game, I had to pay up on my bet. It’s the kind of stunt that makes learning and playing an educational game fun for the students. And I wasn’t alone in it, either, as three of us teachers or leaders licked the same slug (I lucked out and got the middle section, I felt sorry for the lady who got the tail).

Dedication will help make a team, department, organization, or industry successful. By being dedicated enough to teach others about security, you can help make your peers more secure in their computing habits. By having the dedication to chase down that alert you think might be a false positive, but you’re just not sure, you can find the evidence which allows an adversary to be discovered. By coming in at 3:00am because all hell has broken loose, you can help make things right. And, by being dedicated enough to craft to be willing to learn, even on your own time (we all do it), you’re helping to build the capacity for your organization to respond to new challenges.

Click here to download the slides.