Overview

The CBEST process is the regulated approach taken to the routine testing of financial institutions with a suitable presence within the UK. It is now a mature process which makes it a useful guide for basing non-regulated red teaming operations on. For example, aligning any deconfliction and pre-testing arrangements on the CBEST approach.

In this post, I’ll summarise the CBEST implementation guide, trying to highlight the key parts where possible! The full guide is available on the Bank of England site, and is worth the read to pick up additional details where needed!

Some of the acronyms used are detailed in the table below.

Acronym	Meaning
FMI	Financial Market Infrastructures (e.g. Banks)
FCA/PRA	UK Financial Regulators (Financial Conduct Authority and Prudential Regulation Authority)
TISP	Threat Intelligence Service Provider
PTSP	Penetration Testing Service Provider
CG	Control Group (The members of the FMI used to co-ordinate testing)
DRA/DRCA	Detection and Response Capability Assessment (A tabletop exercise to assess the ability of the blue team to respond to and contain the active testing element)
TI	Threat Intelligence
PT	Penetration Testing (Often used to refer to the active red team testing phase)
PID	Project Initiation Document

CBEST Concepts

Triggers

To commence a CBEST test, one of several triggers can be used:

• The regulators (FCA/PRA) requesting a CBEST exercise as part of the regular schedule
• The FMI requesting to perform a CBEST, with regulatory approval
• In response to an event/incident

Stakeholders

There are 4 key groups of stakeholders:

• The regulators
• The control group at the FMI
• The Threat Intelligence Service Provider (TISP)
• The Penetration Testing Service Provider (PTSP)

Regulators

The regulators will perform several steps:

• Oversight of CBEST outcomes and remediation
• Receiving and reacting to high priority outcomes from the TI and PT phases
• Reviewing reports to determine thematic findings

Control Group

The control group will appoint a Control Group Coordinator (CGC) who will oversee the engagement and coordinate activities within the Control Group (CG), approval is required from the regulators before any changes are made to the CG.

The CG must include only the bare minimum required employees to safely run the engagement, typically this would be senior leadership (COO, CISO), as well a senior SME on the applications/systems to be tested. Third parties can be included in the CG, with a separate process detailing that!

The CG has several main responsibilities:

• Ensuring the requirements of the CBEST implementation guide are met
• A project plan is created and updated
• The CBEST process is delivered in a risk controlled manner
• Secrecy of the testing being maintained
• Coordination between the regulators, TISP and PTSP

There are also several smaller tasks which the CG handles:

• Determining a code name/project name for the testing
• Redacting any sensitive information from reports before sharing with the regulators
• Producing a Project Initiation Document (PID), including the CBEST management plan along with specific plans from the TISP and PTSP

Risk Management

The CG needs to make a risk management plan, aiming to mitigate all risks wherever possible in advance of testing beginning. This will require regular updating throughout testing to ensure it remains relevant.

Threat Intelligence Service Provider

The TISP has the following responsibilities:

• Providing an external TI view of the FMI
• Determining realistic threat profiles for the chosen CBPs for use in subsequent stages
• Complete the Threat Intelligence Maturity Assessment (TIMA) on the FMI
• Providing an early view of the TI report to the PTSP to ensure it is satisfactory
• Provide input to the PT phase and the final report as required. 

Penetration Testing Service Provider

The PTSP has the majority of the responsibilities:

• Working with the TISP to ensure the TI report is sufficient from an early stage
• Design and plan the PT phase based on the information provided by the TI report
• Agree a risk management process with the CG
• Perform the red team testing
• Provide regular updates on testing
• Complete the Detection and Response Capability Assessment (DRCA)
• Produce the final report
• Share feedback with the regulator during the debrief call

Phases Overview

There are 4 phases to CBEST:

• Phase 1 (Initiation, 4-6 weeks)
  • Scope is determined by regulators
  • TISP and PTSP approached by CG
  • Consists of:
    • Launch
    • Engagement
    • Scoping
    • Procurement
• Phase 2 (TI Phase, 10 weeks)
  • TISP produces the TI report
  • PTSP takes the TI report and produces a penetration testing plan as a result
  • Consists of:
    • Direction
    • Intelligence
    • Validation
    • TI Assessment
• Phase 3 (PT Phase, 14 weeks)
  • The intelligence-led penetration test is performed
  • DRA and TIMA are produced by the PTSP and TISP respectively
  • Consists of:
    • Planning
    • Execution
    • D&R Capability Assessment (DRCA)
    • Review
• Phase 4 (Closure, 4 weeks)
  • Debrief held between all parties
  • Remediation plan produced by CG and overseen by regulators
  • Consists of:
    • Remediation
    • Debrief
    • Supervision

Initiation

Launch

The regulators will contact the FMI with a notice of the CBEST process beginning, which must ‘start’ within 40 days. During this initiation phase, several items will be decided, such as:

• Aims of the testing
• The roles and responsibilities of the CG
• Configuration of secure transfer solutions/communication platforms
• Contractual and legal requirements
• The members of the CG and their responsibilities 

Engagement

The ‘Engagement’ process is only complete when the following items have been done:

• The ‘Kick Off’ meeting, between the CG and regulators
• The regulator and CG agree the key stakeholders and responsibilities

Scoping

The ‘Scoping’ process involves the identification of IBS/CBP’s in-scope for testing, as well as which section(s) of the CIA triad these systems will be tested against. E.g. IBS123 to be targeted for confidentiality and integrity (CI). 

A scoping meeting is then held between the regulators and the CG to discuss and agree this scope. The CG will respond to this proposal by identifying the key systems which underpin these IBS/CBPs, as well as justifying why the given systems were chosen. 

At this point, third party providers may be identified for inclusion in subsequent testing or being read-in to testing. They need to be aware of the strict secrecy and confidentiality requirements.

Finally, the CG and regulators agree the compromise actions to be taken by the testers. This is then considered within the TI report.

The ‘scoping’ stage is completed when:

• The regulator, CG and CBEST executive/sponsor has approved the ‘Scope Specification Document’
• CG and regulator agree indicative CBEST timeline
• CG conduct risk assessment
• CG produce the PID, which includes:
  • Governance & Risk Assessment
  • Project Plan
  • Connect lists etc

Procurement

At this stage, the CG will engage with approved CBEST providers for both the TISP and PTSP. After determining an appropriate provider, contracts should be exchanged, containing the required legal and privacy clauses from the regulators. The PID is then completed, including the final schedule of meetings for the FMI and regulator.

Testing cannot progress past this point until ‘[it is] confirmed to the regulator that appropriate legal contracts are in place between the firm/FMI and the TISP/PTSPs.’

The PTSP needs to ensure the appropriate permissions and approvals are in place for testing, such as rules of engagement. There are requirements for the CG to notify the regulators if they believe that the testing provider loses their CREST-supplier status or are using unqualified staff during testing, for example they must have a CCTIM, CCRTS and CCRTM qualified testers.

On the flip side, the testing providers must report to the regulator if they believe the testing process is being ‘manipulated’, such as through informing the SOC of testing or by placing pressure on the testing provider to present a more positive view.

TI Phase

Following the Initiation phase, the TI phase starts with the TISP. The PTSP is only included when threat scenarios have been produced. The aim of the TI Phase is to assess the external perimeter of the FMI, produce accurate threat profile(s) for specific threat actors and to help shape the testing performed in the PT phase.

To initiate this phase, the CG can direct the TISP, using information from the FMI’s TI function, so long as the secrecy of testing is preserved. The TI provider then analyses a range of TI sources, presenting it back the regulator and CG during the validation workshop.

This phase also includes a Threat Intelligence Maturity Assessment (TIMA), though this can be completed after the PT phase has concluded to maintain secrecy.

There are 4 steps to this stage:

• Direction
• Intelligence
• Validation
• TI Assessment

Direction

To start this phase, the CG send the CBEST Scope Specification to the TISP, informing them of the systems in scope, it is recommended for this scope to also be sent to the PTSP to prepare them in advance.

Due to the often limited public information or highly bespoke nature of some of the in-scope systems, the CG are expected to provide some supplementary information to the TISP to assist with their research. This is done to create a ‘grey box’ approach to testing, allowing for simulation of testing without requiring the testers to break any legal or ethical considerations. 

Some examples of this information are:

• Organisational structure (Branding, IT service providers, physical locations)
• Business and technical overviews for each in-scope system
• A current FMI TI report, with sources
• Various identifying details (Domains, IP ranges, social media etc)
• Known cyber attacks (Known leaked data, DLP strings)
• Details to identify unknown attacks (Project names, naming conventions etc)

This data is captured in a ‘IBS-focused Threat Intelligence plan’ produced by the TISP. This is sent to the FMI who then distribute to the PTSP.

The PID is then updated, with the regulator being made aware of any key changes to the risks.

Intelligence

During this data gathering stage, 2 key types of data are gathered:

• Targeting (Attack surface)
• Threat intelligence (To develop testing scenarios)

Should a significant vulnerability be discovered, then the TISP must notify the regulators and FMI. The FMI can patch it but may have to grant representative access to the PTSP during the PT phase. It is expected that the TISP minimise their chance of being detected, relying on passive means where possible. 

Targeting

This section is intended to be broadly scoped, under the assumption that a threat actor would likely compromise unrelated infrastructure in order to move laterally towards the targeted systems. This should include a range of details on the attack surfaces of people, processes and infrastructure belonging to the FMI. This can include customer data or leaked information. 

This section results in the ‘Targeting Report’, which feeds into the TI Report. 

Threat Intelligence

This section focuses on key threats to the FMI as a whole, with detailed sections on the greatest threats and scenarios which the threats may look to exploit. This builds upon the data gathered in the Targeting Report, taking any identified assets and including them in the proposed scenarios. If relevant data can’t be found, then a more generic attack is planned.

Each scenario must map to at least 1 of the identified IBS’s, including technical information for a likely scenario:

• Objectives/targets for the attack 
• Information on the threat actors
• TTPs to be used
• The stages of the attack are explained using a framework such as MITRE ATT&CK or the Cyber kill chain

This output is meant to direct the PTSP during the testing, with the following key outcomes:

• Accurate scenario setting and guidance for the PTSP 
• The ‘flags’ to be captured as part of for the various objectives
• A narrative/evidence for the FMI to use to justify remediation and improvement following testing.

Scenario Development

Following development of the TISP, the PTSP will produce a draft penetration test plan. This is the transition between the TISP and the PTSP, and is led by the PTSP.

It is expected that several scenarios may contain common elements for efficiency, but then branch out to various actions on objectives. It is key that all of the testing maps back to the details within the Threat Intelligence Report and the IBS’s to be targeted.

The TISP and PTSP are meant to produce the most relevant path an attacker could take to perform the test actions. This should be done using a visual representation, using standard attack framework terminology (MITRE ATT&CK and Cyber Kill Chain). Using a visual representation helps the CG, FMI and regulator to show the mapping from high level plans through to technical implementation and any defensive controls which may be impacted. By collaboratively producing this document, a blend between threat intelligence accuracy and realism for implementation can be achieved. 

The visual diagrams need to be used by the PTSP as part of their PT update to the regulators.

Malicious insider and supply chain scenarios ‘should always be analysed and discussed during CBEST’. By the PTSP and TISP working closely with the CG, the feasibility of various scenarios can be considered to make a realistic attack plan for the RT stage. This may also include detailed considerations such as the type of account required and access level. The CBEST guidance recommends using staff or third parties to increase realism of the testing. This may require extensive pre-planning to ensure the relevant agreements and NDA’s are in place.

The CBEST process recommends against performing DoS or physical attacks. They also recommend against testing to gain representative access, instead the PTSP should actually perform the action. For example, modifying a production website rather than simply gaining sufficient privileges to do so. They recommend this style of ‘proving acccess’ testing is conducted outside of the CBEST process. This is due to the level of simulation involved in such testing, leading to a lack of real-world outcomes. 

During testing, other attack paths may be identified which again should be tested after the CBEST process has concluded. 

TI Reporting Process

The process of delivering the TI and Targeting report is in several steps:

1. the TISP produces a first draft for delivery to the CG;
2. the CG forwards the draft documents to PTSP;
3. the TISP subsequently holds an Intelligence workshop with the CG and the PTSP to discuss the draft report and obtain feedback;
4. after the Intelligence workshop (between the CG, TISP and PTSP), regulator may ask to hold a mid-point workshop where the TISP presents a summary of the TI assessment.
   1. If a draft of the Targeting Report and Scenarios are available, the workshop can be used to have a preliminary discussion on the TI assessment so far and planning, ahead of the Validation;
5. the TISP produces a revised second draft for delivery to the CG.

Once the CG has received the revised second draft, the following activities take place:

1. The CG forwards the Threat Intelligence Report and Targeting Report to the regulator and/PTSP; both draft and final versions of the Threat Intelligence Report and Targeting Report are sent to the regulator to give them sufficient time (at least one week) to review the reports prior to the Validation workshop.
2. Validation workshop is held.
3. After the Validation workshop the TISP makes any further changes to the two reports and issues the final versions for delivery to the CG which then forwards the documents to the regulator and PTSP.
4. Only when regulator feedback has been incorporated into the Targeting Report and the Threat Intelligence Report, can these be deemed final.
5. Final reports cannot be shared more widely than with the CG, within the participant/firm, until the entire CBEST process has been completed.

Validation

The validation workshop is a 3 hour workshop involving all CBEST stakeholders (CG, TISP, PTSP, regulator). The following activities are performed:

• The TISP presents the Targeting and TI Report
• The regulators provide feedback on both reports
• The PTSP presents the draft Penetration Testing Plan. This includes detail on the mapping of the TI scenarios to IBS’s and the various parts of testing (Risk mitigation plans, escalation procedures, estimated delivery dates and so on)

The TISP then produces a final version of the their reports for the CG, who then forward it onto the PTSP and the regulator. This handover marks the formal handover from TISP to PTSP.

The PTSP should revise their documents to reflect any changes. The CG may need to update their PID based on the details discussed. This may lead to key stakeholders being notified of any key changes (Such as increased/changed risk involved with testing).

Threat Intelligence Maturity Assessment 

The TIMA is part of the TI phase, but is typically conducted after the PT phase to prevent drawing attention to the activity. This feeds into the review workshop along with the DRA. 

The aims of the TIMA are to objectively assess the FMIs cyber security capability, gain greater understanding about the capability of the wider FI sector’s capability and to raise increased awareness within the FMI on its TI capability. 

Relevant, senior employees should be leveraged to perform this assessment, third parties can be included where appropriate if some functions are outsourced. All data collected must be reviewed and validated by a CREST Certified Threat Intelligence Manager (CCTIM).

The process for the TIMA is:

1. The regulator provides the TIMA document and CREST Threat Intelligence Maturity Assessment Tool to the FMI
2. The TISP holds a meeting with the FMI to handover both documents and explain their usage/the wider TIMA process
3. The FMI self-assesses its capability across each of the ‘Capability Indicators’ (CI), along with gathering relevant evidence.
4. A meeting is held between the FMI and the TISP to present evidence and agree final scores
5. The TISP provides the CG and regulator the Intelligence Assessment Report, summarising the findings
6. The outcomes are discussed during the final PT Review Activity, with any recommendations being covered within the CBEST Remediation plan.

Penetration Testing Phase

This phase should take around 14 weeks, delivered by the PTSP. At the beginning of this phase, the PTSP produces a risk management plan to prevent issues from occurring. 

There are 4 steps to this phase:

• Planning
• Execution
• Detection and Response Capability Assessment (DRCA)
• Review

Additionally, the TIMA is also completed, though this is officially within the TI Phase. 

Planning

The planning phase takes the CBEST Scope Specification and both the TI and Targeting report to produce the PT Plan. It is expected that the PT Plan should be able to accurately replicate the various threat actors and TTPs identified within the TI phase. The PTSP needs to ensure a risk management plan is produced to reduce any risks which the FMI could be exposed to during testing. 

It is expected that all scenarios must be suitably tested, if there is not sufficient time during testing then an extension should be considered. This is particularly important for more resource-intensive scenarios where dedicated environments or assets are required, such as simulating a malicious insider.

The final output is the Penetration Test Plan (PT Plan), which is accompanied by the Penetration Test Risk Management Plan, which is delivered to the FMI and the regulator. This document maps the proposed testing back to the scenarios from the TI Report and to the IBS’s from the CBEST Scope Specification. The report should also include the testing plan and attack plan, described using established frameworks such as MITRE ATT&CK or Cyber Kill Chain.

For each step of the kill chain, the following description should be produced:

• Prerequisites to be implemented ahead of the execution of the action
• The target action/flags
• The success criteria or expected result of the actions
• The possible de-chaining actions and criteria to be met to request the information of the CG
• Expected timeline for each de-chaining action

Execution

It is expected that all testing is conducted using live, production assets unless of legal/ethical constraints. Should there be any changes to the scenarios from what was produced in the TI Report, then the CG and regulator as required. Dechained or ‘leg up’ actions are permitted, but must be agreed by the regulator and noted within the PT Report. 

Regular updates must be provided to the regulator and CG as testing progresses. Updates should detail the status of testing, progress towards ‘flags’ and any risks/issues. These meetings can be conducted whenever, but are typically weekly. 

The draft PT report is produced typically within 2 weeks of the end of testing. The report must contain:

• Executive summary for the Board and Senior Executive
• Executive summary for technical leaders (eg COO, CIO, CISO, etc)
• Description of results in relation to the scenario and target actions in scope
• Summary of the findings
• Detailed description of the findings and recommendations for the firm/FMI
• Breakdown of the scenarios, describing the progress made by penetration testers in terms of their journey through the various stages of each threat scenario.

All sensitive/identifiable information must be redacted before sharing with the regulator. This includes names, IPs, server details etc. 

Assessment

Before the final review activity, the Detection and Response Capability Assessment (DRCA) is performed by the PTSP. this also has ‘Capability Indicators’ (CIs) like the TIMA, which are both qualitative and quantitative. This is a similar process to the TIMA, though is focused on the response and SOC capability, as opposed to the TI function. 

The FMI must provide relevant staff members to answer the assessment, with the PTSP providing a CCSAM to evidence and validate the responses. This must be conducted and returned to the regulator no more than 2 weeks after completion of testing. 

Review

The CG, regulators, PTSP and TISP hold a ‘Review Workshop’ to review the draft penetration testing report, DRCA, TI findings and recommendations from the TIMA. These are broken down as follows:

1. PT test performance and identified vulnerabilities (led by the PTSP)
2. Firm/FMI’s D&R capability (led by the PTSP)
3. Review of TI findings and recommendations (led by the TISP)
4. Firm/FMI’s TI maturity assessment (led by the TISP)
5. High-level discussion on mitigating factors and proposed remediation (led by the CG)

For each stage of the testing, the PTSP discusses the progress made and what could have been achieved with more time/resources. Scenarios which were impracticable, unsuitable or not chosen for testing can be discussed for consideration for future testing by the CG/FMI. It will be checked that the scope has been sufficiently covered by the testing. 

The output of this is the final PT Report, for delivery to the FMI, who forward it onto the regulators.

Closure Phase

The final phase of testing leads to the development of the remediation plan. The 3 key phases of this are:

• Remediation
• Debrief
• Supervision

Remediation 

The CG produces a remediation plan in response to the final TI and PT reports, aiming to remediate the issues identified by both assessments, as well as key issues from the DRCA and TIMA. The CG and regulator meet to discuss this draft plan, with feedback given by the regulator to shape the final remediation plan. This final plan is delivered to the regulator.

This final plan should capture the risk and impact assessments from the FMI, showing how they translate into business risk against the IBS’s. The final plan requires input from the board, senior management, risk owners and risk management functions. Technical remediation needs to be agreed with technical leaders and relevant SMEs. The wider findings from the CBEST process can then be shared across the FMI to more generally improve processes and function. 

Debrief

At the end of the CBEST Assessment, representatives from the TISP and PTSP meet with the regulator to discuss:

• What activities and deliverables went well/what could have been improved
• Which aspects of the CBEST process went well/what could have been improved.
• Other miscellaneous feedback

The output of this is logged by the regulator to improve the CBEST process.

Supervision

Following completion of this assessment, the regulator oversees the progress of the remediation plan with regular meetings over a 6-12+ month timespan. The FMI is expected to notify the regulator as remediation actions are closed.

Thematics

After the CBEST process is completed, the regulator may produce thematic analysis of themes across multiple CBEST engagements. The anonymised report is produced by the FCA and PRA alongside the NCSC to improve resilience across non-FI firms. 

Conclusion

Hopefully this summary of the CBEST Implementation Guide was useful! If you are looking to implement this or learn more about specific aspects, then I would recommend reading the full guide, as well as picking up a copy of ‘Red Team Development and Operations: A Practical Guide’ by James Tubberville and Joe Vest, which is an excellent reference!