NERC Committee Agenda Packages

The NERC Board of Trustees is meeting this week, and along with that are several standing committee meetings. While the meetings will not be simulcast online, the agendas for the meetings oftentimes include some interesting reading. A couple items from the Members Representative Committee and the Compliance Committee were interesting.

Members Representative Committee

On page 44 of the MRC agenda package, Compliance Guidance Implementation is discussed. They provide an update on the new process where compliance guidance will be vetted by the “ERO Enterprise,” and after that vetting and approval then the guidance will be given “deference” from auditors in all the regions. This update says that the task force is beginning to review existing documents that can be submitted to the ERO Enterprise for endorsment. More interesting, though, is that the CCC members of the task force are developing a process to approve organizations to be able to submit guidance documents even if the organization is not already on the pre-qualified list to submit guidance documents.

Another interesting bit of news is that a CMEP Practice Guide focused on what it means for auditors to “provide deference” is being developed. The guide on how to provide deference will be the first CMEP Practice Guide published. The CMEP Practice Guides are basically guidance created by the ERO Enterprise which provide direction to auditors on how they should conduct audits.

Compliance Committee

Lessons Learned Documents

This one had this great quote: “The CIP Version 5 Transition Advisory Group identified specific issues with the CIP Version 5 standard language, which were temporarily resolved through Lessons Learned documents.”

It then lists the issues that are being referred to the CIP V5 Revisions Standards Drafting Team:

• Transmission Owner Control Centers
• BES Cyber Assets/Programmable Electronic Devices
• Virtualization
• External Routable Connectivity

To call these issues even temporarily “resolved” is quite the stretch. Virtualization is not addressed at all in the Lessons Learned (LL) documents. While the others were addressed, they were not resolved. For example, the LL on BES Cyber Assets doesn’t provide a definition for “programmable,” which forms the basis of the Cyber Asset definition but doesn’t have any clear definition itself.

Note: A coworker told me that she’s heard the same line (“temporarily resolved through LL documents”) used by the V5 TAG several times. This is the first I’ve noticed it, though.

IRAs and ICEs

The 2015 ERO Enterprise Annual CMEP Report was included in the agenda package. It said that there were 236 entities scheduled for an audit in 2015. The Regions conducted a total of 230 Inherent Risk Assessments for entities on the audit schedule, so they got almost all of them. They also performed 31 Internal Controls Evaluations for entities on the audit schedule, or about 13% of the entities that had an IRA performed. It would have been nice to have a breakdown of those numbers by region. Are all eight regions represented in those 31 ICEs? Or are the numbers dominated by just one or two regions? That would be helpful information to have, although it may be available from other sources, I haven’t researched that question.

Outreach Events Focused on Risk-based CMEP

These figures were included in the CMEP report. It seems weird that ReliabilityFirst would have done almost twice as many events as anybody else, but had the second lowest number of participants. These numbers would mean that RF only had about 9 participants per event, which seems quite low. It makes me wonder if the different regions didn’t all use a standardized definition of what constitutes an “outreach event.”

Shodan and SERC Audits

On SERC’s Open Forum Webinar on Monday, January 25, they had a presentation on the Shodan search engine. I wasn’t able to participate in the webinar, but a co-worker did, and the webinar seemed to be based on this one-page document SERC released on January 4, “Shodan: What SERC Registered Entities Need to Know.” SERC said on the webinar that the SERC audit teams will be using Shodan as part of their audits.

Shodan (www.shodan.io) calls itself the “world’s first search engine for Internet-connected devices.” If you haven’t had a chance to explore Shodan yet, you really should go spend a half-hour or so exploring it and seeing what is possible with it. It’s been in the news over the last couple days because they added a new section that allows paid users to browse Internet-connected webcams. You don’t have to be a paid user to use the site, although the paid accounts do have some added capabilities. Shodan also has a section dedicated to Industrial Control Systems, where a user can get tips on how to find ICS. One of their examples is that if you search for ‘title:”xzeres wind”‘, then you can find wind turbines.

SERC’s one-page is a pretty good overview of what Shodan does. One key point that they make is that when somebody uses Shodan, they aren’t actually contacting the target. Shodan has already gotten the information about what devices are on the Internet, and all the user of Shodan is doing is searching this database. SERC calls out two things which an attacker could use Shodan for. First, if an attacker has a vulnerability they are prepared to use, they could use Shodan to find devices which have that vulnerability. As an example, the first alert on ICS-CERT’s website when I checked it today was ICS-ALERT-15-225-01A for Rockwell Automation 1769-L18ER PLC. This alert talks about there being public proof-of-concept exploit code. If you search Shodan for “1769-L18ER/A”, you find 65 results. An attacker could use that list to find possible targets for an attack.

If you’re unlucky (probably the wrong word, because you’re doing something wrong, not just being unlucky) to have one of the devices somebody is targeting on the list that gets returned to the user, you might get targeted. There isn’t much that knowledge of this type of attack using Shodan can actually do for you, though. It’s probably unrealistic for a company to try to go search for all the devices they have in their plant, and even if they did, almost everythiing they found on Shodan would be a false positive since any one result isn’t likely to be from your specific organization.

The other way that SERC points out that an attacker could search an organization’s domain name. This would be used in a more targeted approach. Using the domain of a large energy company, there were 40 results returned by Shodan.

This led mostly to some 302 Redirects and mail servers whose banner message said that the Shodan crawler was not allowed to access them. There weren’t any any ICS protocols in any of the results, as you can see in the “Top Services” section, so this company seems to be in pretty good shape, at least in terms of what Shodan could see in a simple search.

SERC said that they recommend that,

all its registered entities perform a Shodan search for their own domain name(s). For each device found, consider the following questions:
Is it truly necessary for this device to be external-facing?
Have all applicable security patches been installed on this device?
Have unneeded accounts and old passwords been removed/changed?
What security measures are in place on the device (e.g., anti-virus, host-based IPS, firewall)?

They go on to say that, “the SERC audit team intends to utilize Shodan during the early stages of its CIP audits, performing a search for the audited entity’s internet domain name,” and that the audit team will use the entity-provided list of BES Cyber Assets (they actually call them Critical Cyber Assets, but I’m pretty sure they mean BCA) and that any matches will probably be included in the sample selection for CIP-005 and CIP-007. They finish by saying, “the audited entity will be offered the opportunity to provide evidence the device and its associated network(s) are appropriately configured.”

I think that it’s pretty good advice to, as a defender, consider using Shodan to see if you have anything hanging out on the public Internet. From a commonsense perspective, there may be some sense to an auditor using Shodan in order to find devices which are publicly accessible, and therefore giving them a little extra attention, or at least making sure they aren’t missed. Ideally, though, auditors won’t find any control systems on Shodan, and so the search won’t add much value to the audit.

Cyber Asset Lesson Learned Finalized

NERC’s Standards Committee (SC) approved the Lesson Learned (LL)document on CIP-002-5: BES Cyber Assets on January 21, 2015. This document was originally released for comment on September 9, 2015, and the final document is dated December 7, 2015. It was created by NERC’s V5 Transition Advisory Group (V5TAG).

One of my criticisms of the LL process is that the LLs will be commented on, get revised, and then get approved and posted by the SC without industry having a chance to see or vet the revisions before they are finalized. While NERC has re-posted some of the LLs for comments, they’ve generally only done that when there are major revisions, and they definitely haven’t done so for all (or even most) of the Lessons Learned documents. This leads to a limited number of people actually having input into what the final, official, document contains. I believe this process has the potential to introduce flaws into the documents, and it happened again with the Cyber Asset LL.

The final document will be available from the “Related Documents” section of the CIP-002-5.1 page at NERC’s website, and the consideration of stakeholder comments will be available from the transition program page. The redlined version of the document is a little harder to find, but it is available from the SC agenda package for their January 21, 2016 meeting.

Sidenote: At one point, the redlined versions were not provided by the V5TAG or the SC. As part of my day  job I actually went and created a redlined version from the previous draft and the finalized draft for a couple of the LLs. That was a pain-in-the-butt, so the inclusion of the redlined versions the last few times finalized LLs were released was much appreciated.

Anyways, the purpose of this post is to look at some of what is in the LL and some of what was taken out of it.

When Cyber Assets meet a threshold of BES impact they become BES Cyber Assets (BCA) which may be grouped by responsible entities into BES Cyber Systems (BCS)

While I don’t disagree with this statement, I think it may be misleading to some people, specifically the word “threshold”. If the Cyber Asset has an adverse impact on the BES within 15 minutes of being misused, degraded, or unavailable, then it is a BES Cyber Asset.[^This is a simplified definition, obviously, and there are more qualifiers and requirements, but this summary works for now.] While the 15 minutes could be considered a threshold, there is no allowance in the standards for the adverse impact to have some minimum threshold. The way the sentence in the LL is worded, with “threshold” directly preceding “of BES impact” could cause some confusion for somebody who associates “BES impact” with the “adverse impact” from the BES Cyber Asset definition.

Some study participants assessed each functional system at a site or facility to determine its potential to adversely impact the BES in 15 minutes or less…Other study participants identified all Cyber Assets, grouped them into BES Cyber Systems, and evaluated the impact of the resulting system.

I like how this provides two approaches to achieving the goal of identifying BES Cyber Assets. This is a good example of an LL being written in such a way that it acknowledges that, “there may be other legitimate ways to fulfill the obligations of the requirements that are not expressed within this supporting document,” as the LL puts it in the introductory paragraph.

Did the device’s function directly impact the reliable operation of a BES asset?

The word “directly” was removed from here, and there were similar edits at other places. This was done in response to a comment from ACES, and seems like a good decision, although it leads to a problem discussed next. There’s nothing in the definition of BES Cyber Asset that talks about a difference between “direct” and “indirect” impacts, and so adding that concept here seems like it would add to confusion insteading of making things more clear.

Did the device function as an EACMS, PACS, or Intermediate System? These types of devices, such as firewalls, intrusion prevention systems or physical access controllers and others that perform security functions may indeed have an adverse impact if they are unavailable or misused to allow unauthorized access or deny authorized electronic or physical access, but it is an indirect impact. These devices have their own definitions and requirements in the CIP version 5 Reliability Standards and therefore are not considered BCAs due solely to their impact.

This is the most glaring example in this document of the problem of not having LLs vetted by industry after being edited. I do not think these changes would have made it through a public vetting process, if there had been one. In the first part of this paragraph, they are saying straight-up that firewalls, IPSs, or other security devices can have an adverse iimpact if they are unavailable or misused. While these devices do have their own definitions, there is nothing in the definition of BES Cyber Asset that would imply that an entire class of devices can be categorically excluded from the requirement to determine whether they are BES Cyber Assets. By removing those last five words from this paragraph in the draft LL, the V5TAG has gone beyond providing an example of a compliant approach, and has gone into the terrority of interpreting the standards.

There are programmable devices that may have an impact, but have no way to change their executing code, have no concept of a user or authentication, have no ports/services, have no network connectivity, no concept of event logs or alerting, no patches or updates, andor are located in areas where physical security perimeters cannot be established. For these devices, the ERO will allow (for the purposes of assessing compliance with the standards) Responsible Entities to only protect those devices to the extent capable.

The last sentence in this excerpt had, by far, the most comments related to it. EEI went so far as to say that unless the sentence was removed they couldn’t support the document being submitted to the SC for approval. The problem with the sentence was that it implies that non-programmable devices, if they would have an adverse impact within 15 minutes of their misuse, degradation, or unavailability, would have to be protected to the extent capable. The definition of Cyber Asset, though, explicitly states that a Cyber Asset is programmable. That means that a number of new devices would now be in-scope for the CIP standards.

They changed this table to say they were identifying Cyber Assets typically “evaluated” as BES Cyber Assets instead of saying they were typically “identified” as BCA. This change seems to be more in line with what the purpose of the list of device types originally was.

The transition study participants found a lack of clarity in the CIPV5 Reliability Standards because they do not specifically define the “programmable electronic device” component of the BES Cyber Asset NERC Glossary term. Consequently, the CIP V5 Transition Advisory Group referred the identified issues to be evaluated for standards development.

This was added to the document. While a better definitio of Cyber Asset or a definition of “programmable” is needed, unfortunately the creation of an SDT will not be a very quick process, so that clarification likely won’t help most entities very much in the near- to mid-term future.

Another Example of Why Prescriptive Regulations Fail

Last week, the Department of Homeland Security’s ICS-CERT released an advisory for the Advantech WebAccess product. The advisory detailed 15 different vulnerabilities, reported by at least seven different security researchers. They said that the vulnerabilities could be exploited remotely, and that an attacker with “low skill” would be able to eploit the vulnerabilities.

Patrick Coyle pointed out that the release notes and press release for this update, released by Advantech in late December, don’t have any indication that the update has any security-related patches. This, despite having 15 different vulnerabilities being fixed!

For people who are lucky enough to live in the NERC CIP world, this creates an interesting situation. Organizations are required to monitor a “patch source” for cyber security patches. This is likely to be the vendor, in this case Advantech. (Note: I don’t want it to sound like I’m picking on Advantech. The same situation happens with other vendors, too.) If the patch isn’t security-related, then the organization is not required to apply the patch. If the patch is security-related, then they either need to apply the patch or else create a mitigation plan for how they will prevent the vulnerability from being exploited.

Since the Advantech release notes didn’t say anything about this patch being security-related, the organization would have stopped there. Now, a month later, ICS-CERT releases their advisory and we found out, “Oh, that patch was security-related.” Here’s the kicker, though: The organization isn’t required to scour the Internet looking for vulnerability reports. They aren’t required to monitor ICS-CERT, they’re not required to monitor Full Disclosure, nothing. They did what they were required to do when they checked Advantech’s release notes and didn’t find anything that said it was security-related.

So now, we’re in a situation where the organization has a system with at least 15 publicly disclosed vulnerabilities. For the CIP patch management process, they did what they were supposed to do. The compliance people checked to make sure that they evaluated the Advantech patch. The operations people have a product they’re using, and if it’s working, they aren’t going to want to apply an update. The security people may or may not exist, and if they do exist, they may or may not be monitoring ICS-CERT (although I hope they would be).

The bottom line is, you have an entity which is fully compliant with the NERC CIP regulations, which have regulations in place requiring you to apply security patches, yet this security patch hasn’t been applied.

Ted Koppel’s Lights Out is Ridiculous

Ted Koppel released a book, “Lights Out: A Cyberattack, A Nation Unprepared, Surviving the Aftermath,” A couple weeks ago. The book was not received all that well by the electric industry. I haven’t read the book yet, mainly because I don’t want to add money to the sales figures. I’m sure I could find it somewhere online on that Deep, Dark Web, but I generally don’t pirate things just on principle.

He has given several interviews as part of his book tour, though. Among them, one was on the Diane Rehm Show, which has the transcript available here, another was with Hugh Hewitt, with the transcript here, and one with CSO Online available here.

Koppel (from Rehm Show): as great as the tragedy of Paris is, it’s still a conventional form of terrorism that tragically we have grown accustomed to over the years. And what worries me far more is if a group like ISIS, which already has a cyber war capability, and is already probing on the cyber front, they will have the capacity to reach out from wherever they are in the world.

Right at the start, I have to question his terminology. ISIS has not shown any “cyber war” capabilities. This grossly hyperbolizes what ISIS has done. While their propaganda machine uses technology is has been successful, that does not mean that they have a “cyber war capability.”

Koppel (from Rehm Show): There is no way – I mean, the notion that we can keep out terrorists when the terrorists end up being 19 and 20 years old and if you ran into them in the street, you know, unless you can look into their hearts and minds and see what’s going on there, how can you tell?

This is one point where he does make sense. One of the problems with the “War on Terror” idea is that the U.S. tried to fight a war against an idea by using physical troops and weapons. Unless you’re willing to simply exterminate everybody who holds, or likely might hold, the view you disagree with, that isn’t going to work.

Koppel (from Rehm Show): And what ISIS is successfully doing is imbuing in us a sense of fear and suspicion of all Muslims, which is just going to be devastating to the Muslim community, but also devastating to us.

Koppel has talked and been around politics for a long time. When he’s talking about that, he actually seems to be making sense. This is one of the problems with the “Keep all Muslims out of the US” idea.

Koppel (from Rehm Show): We have become now so dependent upon so many different aspects of the internet, that we fail to see that the internet has now become a weapon mass destruction.

And now he starts to go off the rails. To compare the Internet to a WMD is ridiculous.

Koppel (from Rehm Show): They have plans for every possible natural disaster, but the plan is get yourself a two to three supply of food, make sure you have a radio with adequate batteries. Make sure you have flashlights. Make sure you have water and enough medicine to take care of you for two or three days.

I’ve been to presentations by disaster recovery and emergency preparedness professionals. This does not describe what they do. The problem is, the level of follow-through by people in the public is not that great. They give smaller advice like this because they know that if they have this level of advice, people might follow throug on some of it. If you tell them to have a month’s worth of food ready, most people aren’t going to do anything because they don’t know where to start for a problem of that scale.

Koppel (from Rehm Show): the chamber of commerce has done such an effective job of blocking cyber regulation because there are concerns the power industry is deregulated. On one level, that’s good for all of us because it brings the price of electricity down. But that deregulation means it’s not subject to, as the term implies, it’s not subject to very much federal regulation.

This is disingenuous, at best. Koppel’s book is about the Bulk Electric System (BES) going down, not just an outage at some municipal power company or something. And the BES does have cybersecurity regulations, as well as reliability regulations. There may be some debate over how good they are, but it is regulated.

Koppel (from Rehm Show): the deputy director of FEMA is a former vice admiral in the Coast Guard – a very nice man and couldn’t have been more gracious to me – but he said, Ted, I just think you’re wrong. I don’t think this is going to happen. I don’t think we’re vulnerable to this kind of an attack.

…taking out, let’s say, the City of New York? What would you do? Well, he said, I’d have to evacuate.

The next day, I went to see his boss, the administrator of FEMA. Yes, he said, he is absolutely convinced that this can happen and very likely will happen. Well, I said, what happens if the attack targets New York City? Do you evacuate? Oh, no, he said. You can’t evacuate New York City. Too many people. Where are you going to put them? Now, here are the two top people at FEMA in total disagreement, A, about whether it can happen and, B, about what you would do if it does.

This is completely believable (and I’m not being sarcastic). Isn’t bueracracy great?

Rehm: But isn’t NERC the National Electric Regulatory Commission?

Koppel (from Rehm Show): No, NERC is actually the industry body.

I don’t think most people in industry would take this view. The short version is, NERC (the North American Electric Reliability Corporation) is tasked by FERC (Federal Energy Regulatory Commission) with writing and enforcing the rules. Whenever NERC writes the rules, though, they have to be approved by FERC before going into effect.

Koppel (from Rehm Show): But in the final analysis what journalism ought to be about is alerting the public to potential danger.

This is the problem with journalism. Journalism should be about informing and educating. When you view your job as reporting on every threat and every possible danger is when people get an unrealistic view of what dangers they actually face in life. It’s why more people are scared to fly in a plane than ride in a car, even though airplane travel is demonstrably safer.

Koppel (from Rehm Show): There are 3,200 companies out there and several hundred of them are not that big, they’re small. They’re not that wealthy. They don’t have the money to spend on cybersecurity. And if you can get into one of those – and that’s pretty easy – then nations like the Chinese and the Russians have learned how to trace it all the way back into the central SCADA systems.

I’m not even sure what he means by this. Is he saying that if an attacker gets a small rural cooperative distribution company, they can then get easy access to controlling the largest high-voltage lines? Because that’s ridiculous.

Hewitt: I go back to the Durkovich interview. I went and looked her up after I read Lights Out, and I’m sure she’s very competent. She graduated from Duke University in 1994, and she’s done a lot of interesting things. But the military people you talk with who are more my age, 59, or have been out of command for a couple of years versus the youngsters, they’re just much more sober about this, realistic about this.

I don’t think that being older leads to a better understanding of the cybersecurity field. Realistically, I don’t think that matters a bit, other than to say that people who are retired, formerly high-level government workers aren’t necessarily who I’d be asking for a current assessment of how things are.

Koppel (From Hewitt Show): And frankly, I’m not an expert on it today. I am simply, as I have been doing for most of my professional life, reporting what people who know more about a subject than I do tell me.

We’ll come back to this one. Just remember, it’s important to pay attention to who you decide to interview when you realize you don’t know about something and want to write a book on it.

Hewitt: There’s no more chilling anecdote than at the 2011 Black Hat conference, Ted Koppel, where you say someone stood up and gave out the password to all of the SCADA’s.

Koppel: Yup, that’s right.

I’m pretty sure he’s talking about this presentation by Dillon Beresford. This literally makes no sense, though. That Koppel would just agree with something as ridiculous as giving out the “the password to all of the SCADA’s” is almost beyond belief. I know he said he wasn’t an expert, but if you’re going to write a book on it, you need to at least realize that doesn’t make sense.

And then there’s the best interview response of them all.

CSO Online: Did you interview penetration testers who have experience in the electric generation/transmission sector for this book?

Koppel: No, I did not.

He writes an entire book, brags about spending a year-and-a-half on it and how many people he’s interviewed, and he didn’t bother to even ask one person who actually has experience in doing what he says is going to happen? If you want to be educated on hacking the power grid, maybe you should talk to somebody who actually knows how to hack SCADA systems.

Hewitt: I would say on Pages 96–99 is perhaps the most, an account of the most disturbing interview I’ve read in a long time. You’re sitting down, and all honor to Jeh Johnson, the Secretary of Homeland Defense, he’s a public servant and a good man, but when you sit down and you talk to him about the threats to the power grid, I quote here, “Johnson’s answer ran slightly more than 13 minutes, and he never addressed the question. It was,” you concluded a little later, “not an area in which he had any expertise.”

I think that last sentence could just as easily have been applied to Ted Koppel, himself.