NERC Committee Agenda Packages

The NERC Board of Trustees is meeting this week, and along with that are several standing committee meetings. While the meetings will not be simulcast online, the agendas for the meetings oftentimes include some interesting reading. A couple items from the Members Representative Committee and the Compliance Committee were interesting.

Members Representative Committee

On page 44 of the MRC agenda package, Compliance Guidance Implementation is discussed. They provide an update on the new process where compliance guidance will be vetted by the “ERO Enterprise,” and after that vetting and approval then the guidance will be given “deference” from auditors in all the regions. This update says that the task force is beginning to review existing documents that can be submitted to the ERO Enterprise for endorsment. More interesting, though, is that the CCC members of the task force are developing a process to approve organizations to be able to submit guidance documents even if the organization is not already on the pre-qualified list to submit guidance documents.

Another interesting bit of news is that a CMEP Practice Guide focused on what it means for auditors to “provide deference” is being developed. The guide on how to provide deference will be the first CMEP Practice Guide published. The CMEP Practice Guides are basically guidance created by the ERO Enterprise which provide direction to auditors on how they should conduct audits.

Compliance Committee

Lessons Learned Documents

This one had this great quote: “The CIP Version 5 Transition Advisory Group identified specific issues with the CIP Version 5 standard language, which were temporarily resolved through Lessons Learned documents.”

It then lists the issues that are being referred to the CIP V5 Revisions Standards Drafting Team:

• Transmission Owner Control Centers
• BES Cyber Assets/Programmable Electronic Devices
• Virtualization
• External Routable Connectivity

To call these issues even temporarily “resolved” is quite the stretch. Virtualization is not addressed at all in the Lessons Learned (LL) documents. While the others were addressed, they were not resolved. For example, the LL on BES Cyber Assets doesn’t provide a definition for “programmable,” which forms the basis of the Cyber Asset definition but doesn’t have any clear definition itself.

Note: A coworker told me that she’s heard the same line (“temporarily resolved through LL documents”) used by the V5 TAG several times. This is the first I’ve noticed it, though.

IRAs and ICEs

The 2015 ERO Enterprise Annual CMEP Report was included in the agenda package. It said that there were 236 entities scheduled for an audit in 2015. The Regions conducted a total of 230 Inherent Risk Assessments for entities on the audit schedule, so they got almost all of them. They also performed 31 Internal Controls Evaluations for entities on the audit schedule, or about 13% of the entities that had an IRA performed. It would have been nice to have a breakdown of those numbers by region. Are all eight regions represented in those 31 ICEs? Or are the numbers dominated by just one or two regions? That would be helpful information to have, although it may be available from other sources, I haven’t researched that question.

Outreach Events Focused on Risk-based CMEP

These figures were included in the CMEP report. It seems weird that ReliabilityFirst would have done almost twice as many events as anybody else, but had the second lowest number of participants. These numbers would mean that RF only had about 9 participants per event, which seems quite low. It makes me wonder if the different regions didn’t all use a standardized definition of what constitutes an “outreach event.”

Shodan and SERC Audits

On SERC’s Open Forum Webinar on Monday, January 25, they had a presentation on the Shodan search engine. I wasn’t able to participate in the webinar, but a co-worker did, and the webinar seemed to be based on this one-page document SERC released on January 4, “Shodan: What SERC Registered Entities Need to Know.” SERC said on the webinar that the SERC audit teams will be using Shodan as part of their audits.

Shodan (www.shodan.io) calls itself the “world’s first search engine for Internet-connected devices.” If you haven’t had a chance to explore Shodan yet, you really should go spend a half-hour or so exploring it and seeing what is possible with it. It’s been in the news over the last couple days because they added a new section that allows paid users to browse Internet-connected webcams. You don’t have to be a paid user to use the site, although the paid accounts do have some added capabilities. Shodan also has a section dedicated to Industrial Control Systems, where a user can get tips on how to find ICS. One of their examples is that if you search for ‘title:”xzeres wind”‘, then you can find wind turbines.

SERC’s one-page is a pretty good overview of what Shodan does. One key point that they make is that when somebody uses Shodan, they aren’t actually contacting the target. Shodan has already gotten the information about what devices are on the Internet, and all the user of Shodan is doing is searching this database. SERC calls out two things which an attacker could use Shodan for. First, if an attacker has a vulnerability they are prepared to use, they could use Shodan to find devices which have that vulnerability. As an example, the first alert on ICS-CERT’s website when I checked it today was ICS-ALERT-15-225-01A for Rockwell Automation 1769-L18ER PLC. This alert talks about there being public proof-of-concept exploit code. If you search Shodan for “1769-L18ER/A”, you find 65 results. An attacker could use that list to find possible targets for an attack.

If you’re unlucky (probably the wrong word, because you’re doing something wrong, not just being unlucky) to have one of the devices somebody is targeting on the list that gets returned to the user, you might get targeted. There isn’t much that knowledge of this type of attack using Shodan can actually do for you, though. It’s probably unrealistic for a company to try to go search for all the devices they have in their plant, and even if they did, almost everythiing they found on Shodan would be a false positive since any one result isn’t likely to be from your specific organization.

The other way that SERC points out that an attacker could search an organization’s domain name. This would be used in a more targeted approach. Using the domain of a large energy company, there were 40 results returned by Shodan.

This led mostly to some 302 Redirects and mail servers whose banner message said that the Shodan crawler was not allowed to access them. There weren’t any any ICS protocols in any of the results, as you can see in the “Top Services” section, so this company seems to be in pretty good shape, at least in terms of what Shodan could see in a simple search.

SERC said that they recommend that,

all its registered entities perform a Shodan search for their own domain name(s). For each device found, consider the following questions:
Is it truly necessary for this device to be external-facing?
Have all applicable security patches been installed on this device?
Have unneeded accounts and old passwords been removed/changed?
What security measures are in place on the device (e.g., anti-virus, host-based IPS, firewall)?

They go on to say that, “the SERC audit team intends to utilize Shodan during the early stages of its CIP audits, performing a search for the audited entity’s internet domain name,” and that the audit team will use the entity-provided list of BES Cyber Assets (they actually call them Critical Cyber Assets, but I’m pretty sure they mean BCA) and that any matches will probably be included in the sample selection for CIP-005 and CIP-007. They finish by saying, “the audited entity will be offered the opportunity to provide evidence the device and its associated network(s) are appropriately configured.”

I think that it’s pretty good advice to, as a defender, consider using Shodan to see if you have anything hanging out on the public Internet. From a commonsense perspective, there may be some sense to an auditor using Shodan in order to find devices which are publicly accessible, and therefore giving them a little extra attention, or at least making sure they aren’t missed. Ideally, though, auditors won’t find any control systems on Shodan, and so the search won’t add much value to the audit.

Cyber Asset Lesson Learned Finalized

NERC’s Standards Committee (SC) approved the Lesson Learned (LL)document on CIP-002-5: BES Cyber Assets on January 21, 2015. This document was originally released for comment on September 9, 2015, and the final document is dated December 7, 2015. It was created by NERC’s V5 Transition Advisory Group (V5TAG).

One of my criticisms of the LL process is that the LLs will be commented on, get revised, and then get approved and posted by the SC without industry having a chance to see or vet the revisions before they are finalized. While NERC has re-posted some of the LLs for comments, they’ve generally only done that when there are major revisions, and they definitely haven’t done so for all (or even most) of the Lessons Learned documents. This leads to a limited number of people actually having input into what the final, official, document contains. I believe this process has the potential to introduce flaws into the documents, and it happened again with the Cyber Asset LL.

The final document will be available from the “Related Documents” section of the CIP-002-5.1 page at NERC’s website, and the consideration of stakeholder comments will be available from the transition program page. The redlined version of the document is a little harder to find, but it is available from the SC agenda package for their January 21, 2016 meeting.

Sidenote: At one point, the redlined versions were not provided by the V5TAG or the SC. As part of my day  job I actually went and created a redlined version from the previous draft and the finalized draft for a couple of the LLs. That was a pain-in-the-butt, so the inclusion of the redlined versions the last few times finalized LLs were released was much appreciated.

Anyways, the purpose of this post is to look at some of what is in the LL and some of what was taken out of it.

When Cyber Assets meet a threshold of BES impact they become BES Cyber Assets (BCA) which may be grouped by responsible entities into BES Cyber Systems (BCS)

While I don’t disagree with this statement, I think it may be misleading to some people, specifically the word “threshold”. If the Cyber Asset has an adverse impact on the BES within 15 minutes of being misused, degraded, or unavailable, then it is a BES Cyber Asset.[^This is a simplified definition, obviously, and there are more qualifiers and requirements, but this summary works for now.] While the 15 minutes could be considered a threshold, there is no allowance in the standards for the adverse impact to have some minimum threshold. The way the sentence in the LL is worded, with “threshold” directly preceding “of BES impact” could cause some confusion for somebody who associates “BES impact” with the “adverse impact” from the BES Cyber Asset definition.

Some study participants assessed each functional system at a site or facility to determine its potential to adversely impact the BES in 15 minutes or less…Other study participants identified all Cyber Assets, grouped them into BES Cyber Systems, and evaluated the impact of the resulting system.

I like how this provides two approaches to achieving the goal of identifying BES Cyber Assets. This is a good example of an LL being written in such a way that it acknowledges that, “there may be other legitimate ways to fulfill the obligations of the requirements that are not expressed within this supporting document,” as the LL puts it in the introductory paragraph.

Did the device’s function directly impact the reliable operation of a BES asset?

The word “directly” was removed from here, and there were similar edits at other places. This was done in response to a comment from ACES, and seems like a good decision, although it leads to a problem discussed next. There’s nothing in the definition of BES Cyber Asset that talks about a difference between “direct” and “indirect” impacts, and so adding that concept here seems like it would add to confusion insteading of making things more clear.

Did the device function as an EACMS, PACS, or Intermediate System? These types of devices, such as firewalls, intrusion prevention systems or physical access controllers and others that perform security functions may indeed have an adverse impact if they are unavailable or misused to allow unauthorized access or deny authorized electronic or physical access, but it is an indirect impact. These devices have their own definitions and requirements in the CIP version 5 Reliability Standards and therefore are not considered BCAs due solely to their impact.

This is the most glaring example in this document of the problem of not having LLs vetted by industry after being edited. I do not think these changes would have made it through a public vetting process, if there had been one. In the first part of this paragraph, they are saying straight-up that firewalls, IPSs, or other security devices can have an adverse iimpact if they are unavailable or misused. While these devices do have their own definitions, there is nothing in the definition of BES Cyber Asset that would imply that an entire class of devices can be categorically excluded from the requirement to determine whether they are BES Cyber Assets. By removing those last five words from this paragraph in the draft LL, the V5TAG has gone beyond providing an example of a compliant approach, and has gone into the terrority of interpreting the standards.

There are programmable devices that may have an impact, but have no way to change their executing code, have no concept of a user or authentication, have no ports/services, have no network connectivity, no concept of event logs or alerting, no patches or updates, andor are located in areas where physical security perimeters cannot be established. For these devices, the ERO will allow (for the purposes of assessing compliance with the standards) Responsible Entities to only protect those devices to the extent capable.

The last sentence in this excerpt had, by far, the most comments related to it. EEI went so far as to say that unless the sentence was removed they couldn’t support the document being submitted to the SC for approval. The problem with the sentence was that it implies that non-programmable devices, if they would have an adverse impact within 15 minutes of their misuse, degradation, or unavailability, would have to be protected to the extent capable. The definition of Cyber Asset, though, explicitly states that a Cyber Asset is programmable. That means that a number of new devices would now be in-scope for the CIP standards.

They changed this table to say they were identifying Cyber Assets typically “evaluated” as BES Cyber Assets instead of saying they were typically “identified” as BCA. This change seems to be more in line with what the purpose of the list of device types originally was.

The transition study participants found a lack of clarity in the CIPV5 Reliability Standards because they do not specifically define the “programmable electronic device” component of the BES Cyber Asset NERC Glossary term. Consequently, the CIP V5 Transition Advisory Group referred the identified issues to be evaluated for standards development.

This was added to the document. While a better definitio of Cyber Asset or a definition of “programmable” is needed, unfortunately the creation of an SDT will not be a very quick process, so that clarification likely won’t help most entities very much in the near- to mid-term future.