HTB Sherlock: Nubilum-1

Nublium-1 is all about cloud forensics, specifically a compromised AWS account that leads to multiple EC2 VM instances, including one acting as a PoshC2 server. I’ll work through the CloudTrail logs in a Splunk instance (run via Docker with video on setup), as well as CatScale logs and other forensic collection to show where the threat actor got credentials for the account, what they did in the cloud, and even identify a victim machine.

Challenge Info

Name	Nubilum-1 Play on HackTheBox
Release Date	30 November 2023
Retire Date	21 March 2024
Difficulty	Medium
Category	Cloud
Creator	sebh24

Background

Scenario

Our cloud administration team recently received a warning from Amazon that an EC2 instance deployed in our cloud environment is being utilised for malicious purposes. Our sysadmin team have no recollection of deploying this EC2, however do recall logging onto the AWS console and finding in excess of 6 EC2s running, which he did not recall deploying.

You have been provided with the interview with our Cloud sysadmin within the triage.

Notes from the scenario:

EC2 (virtual machine in the cloud) is being abused.
Our team didn’t deploy machine, or any of the 6 VMs running.
I’m given an interview with the admin.

Interview

One of the files in the data is interview.txt, and is important to understanding the remaining data.

The interview is as follows:

Incident Responder: Can you tell me about the security measures you took after discovering the EC2 instances being launched randomly?

Cloud System Administrator: Well, I deleted most of the running instances, with exception of the one potentially used for malicious purposes. I didn’t implement any other security measures.

Incident Responder: Did you consider implementing any security measures such as multi-factor authentication or IP whitelisting to prevent similar incidents from happening in the future?

Cloud System Administrator: No, I didn’t think about implementing any additional security measures at that time. I was focused on stopping the instances from launching and deleting the auto-scaling group.

Incident Responder: I see. Can you also tell me about the S3 bucket you mentioned earlier? What kind of files do you store in it?

Cloud System Administrator: Yes, we use the S3 bucket to store a variety of files including backups, static websites, and scripts. It’s an important part of our infrastructure and we access it frequently.

Incident Responder: Where do you usually access either the AWS managemet console or AWS APIs from?

Cloud System Administrator: We work from home and aren’t allowed to have any corporate VPNs on our corporate machines. We usually utilise any 86.5.206.X address.

Incident Responder: Have you implemented any security measures for the S3 bucket?

Cloud System Administrator: No, we haven’t. We want to ensure we can all access this bucket globally without auth.

Incident Responder: I’d like you to work with our team to collect CloudTrail data and also a Phase 1 UNIX collection on the EC2 that may have been used for malicious purposes. Thank you.

Cloud System Administrator: We have provided you the S3 bucket directories in /forela-storage/ & the P1 & CloudTrail collections.

Notes:

Deleted all EC2 instances but the one that was being maliciously used immediately on discovery.
Have not put any extra measures in place (2FA / IP allow-listing).
S3 bucket has backups, websites, scripts and can be accessed globally without auth.
Collecting CloudTrail logs as well as “Phase 1 UNIX” collection on the malicious EC2 instance.

Data

There’s a ton of data:

oxdf@hacky$ unzip -l nubilum_1_int.zip
Archive:  nubilum_1_int.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
...[snip]...
---------                     -------
648774035                     13623 files

Decompressing it gives the interview and four types of artifacts:

oxdf@hacky$ ls nubilum_1_int
catscale_ip-172-31-24-20-20230130-1628.tar.gz  CloudTrail  forela-storage  interview.txt  unusual-directory.zip

Cat-Scale logs - Information about what processes were run on a Linux system. This is part of the “Phase 1 UNIX collection” from the interview.
CloudTrail Logs - Amazon cloud asset logs.
The contents of the exposed S3 bucket (forela-storage).
A directory found on the malicious VM. Also a part of the “Phase 1 UNIX collection”.

Artifact Background

Cat-Scale

Cat-Scale is an incident response collection tool that pulls forensic data from a running Linux host via a Bash script that runs completely with native binaries.

To get to the output, I’ll decompress the archive and look at the structure it created:

oxdf@hacky$ tar -xzf catscale_ip-172-31-24-20-20230130-1628.tar.gz 
oxdf@hacky$ ls catscale_out/
Docker  ip-172-31-24-20-20230130-1628-console-error-log.txt  Logs  Misc  Persistence  Podman  Process_and_Network  System_Info  User_Files  Virsh

The post linked above does a nice job breaking down the different types of data:

Logs - Full copy of /var/log as well as the output of last, utmpdump, w, and who.
Persistence - Attempts to capture main persistence mechanisms used by threat actors.
Process_and_Network - Running processes, SHA1 hashes of all binaries, links to open descriptors, memory maps, and netstat / ss data.
System_Info - System information as well as key and recently changed /etc files.
User_Files - All hidden files in /home.
Misc - Potential webshells, timeline data, and other interesting collection.
Docker / Podman / Virsh - virtualization-related data.

CloudTrail

CloudTrail is:

an AWS service that helps you enable operational and risk auditing, governance, and compliance of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.

This is how AWS does logging. There are a bunch of different CloutTrail Event Types.

There are two types of log files in this collection:

oxdf@hacky$ ls CloudTrail/949622803460/
CloudTrail  CloudTrail-Digest

The Digest file are not useful to use at this time, so I’ll focus on the CloudTrail folder. There’s over eight thousand log files in this directory, all .json files:

oxdf@hacky$ find CloudTrail/949622803460/CloudTrail -type f | wc -l
8681
oxdf@hacky$ find CloudTrail/949622803460/CloudTrail -name '*.json' | wc -l
8681

Tools

Most of this data I can handle from the Linux command line using Bash, text editing, grep, and other tools. To process the CloudTrail logs, I’m going to spin up an instance of Splunk. I’ll show how to make a Splunk Docker container and populate it with this data in this video:

AWS Account Compromise

S3 Bucket Triage

To see what was exposed, I’ll start in the files that were publicly available on the Forela S3 bucket. forela-storage has one folder, backup, which has:

oxdf@hacky$ ls
ec2.py  forela-cytopia-ansible.tar  forela-static-site-version1.tar  forela-static-site-version2.tar  forela-static-site-version3.tar  jenkins-forela-automation.tar  test.txt

test.txt is empty.

All of the .tar files are Docker images. For example:

oxdf@hacky$ tar tf forela-static-site-version1.tar 
31cd39dc731a382a1c525db3508ee53a52f29059ad02c1819363d9e4ea4723a5/
31cd39dc731a382a1c525db3508ee53a52f29059ad02c1819363d9e4ea4723a5/VERSION
31cd39dc731a382a1c525db3508ee53a52f29059ad02c1819363d9e4ea4723a5/json
31cd39dc731a382a1c525db3508ee53a52f29059ad02c1819363d9e4ea4723a5/layer.tar
3897cc63ef89526dccef521a40485962a1b9ad618f7f996f5c7746dcdeca326f/
3897cc63ef89526dccef521a40485962a1b9ad618f7f996f5c7746dcdeca326f/VERSION
3897cc63ef89526dccef521a40485962a1b9ad618f7f996f5c7746dcdeca326f/json
3897cc63ef89526dccef521a40485962a1b9ad618f7f996f5c7746dcdeca326f/layer.tar
551c5675a936ced73e6a206122451e18026db83f2fd04f48dc06a75caadb1323.json
7c46fd33f248d30476f03bb5e6a97155cdfff9e931d349377d3f3f6bbc0b23c7/
7c46fd33f248d30476f03bb5e6a97155cdfff9e931d349377d3f3f6bbc0b23c7/VERSION
7c46fd33f248d30476f03bb5e6a97155cdfff9e931d349377d3f3f6bbc0b23c7/json
7c46fd33f248d30476f03bb5e6a97155cdfff9e931d349377d3f3f6bbc0b23c7/layer.tar
aac21bbe1239c6da7f2bddf4daa80b2fa11fc7eee4c9f9d974560e3b7da22f13/
aac21bbe1239c6da7f2bddf4daa80b2fa11fc7eee4c9f9d974560e3b7da22f13/VERSION
aac21bbe1239c6da7f2bddf4daa80b2fa11fc7eee4c9f9d974560e3b7da22f13/json
aac21bbe1239c6da7f2bddf4daa80b2fa11fc7eee4c9f9d974560e3b7da22f13/layer.tar
beb4c267210679bd6d4bd1e56fdca329f54fcbc9a709764dbfbf3452ba68a1ab/
beb4c267210679bd6d4bd1e56fdca329f54fcbc9a709764dbfbf3452ba68a1ab/VERSION
beb4c267210679bd6d4bd1e56fdca329f54fcbc9a709764dbfbf3452ba68a1ab/json
beb4c267210679bd6d4bd1e56fdca329f54fcbc9a709764dbfbf3452ba68a1ab/layer.tar
d388b754d3eca0d520ee34aa44b5e84d8a892b963df16fbdcd7ddb73d1288c27/
d388b754d3eca0d520ee34aa44b5e84d8a892b963df16fbdcd7ddb73d1288c27/VERSION
d388b754d3eca0d520ee34aa44b5e84d8a892b963df16fbdcd7ddb73d1288c27/json
d388b754d3eca0d520ee34aa44b5e84d8a892b963df16fbdcd7ddb73d1288c27/layer.tar
e3e05b8bbe6a9a3bc5d25fd3f47668152afd75ba05ea77fecb6ea730f67cd622/
e3e05b8bbe6a9a3bc5d25fd3f47668152afd75ba05ea77fecb6ea730f67cd622/VERSION
e3e05b8bbe6a9a3bc5d25fd3f47668152afd75ba05ea77fecb6ea730f67cd622/json
e3e05b8bbe6a9a3bc5d25fd3f47668152afd75ba05ea77fecb6ea730f67cd622/layer.tar
f2210dd9ce0c863a057efdf73b359923001a1f957d167f8b55aa768a36b1dda7/
f2210dd9ce0c863a057efdf73b359923001a1f957d167f8b55aa768a36b1dda7/VERSION
f2210dd9ce0c863a057efdf73b359923001a1f957d167f8b55aa768a36b1dda7/json
f2210dd9ce0c863a057efdf73b359923001a1f957d167f8b55aa768a36b1dda7/layer.tar
manifest.json
repositories

This structure represents the different layers in an image.

ec2.py is a (broken) Python script.

ec2.py

Script

This script is a (broken) Python script that intends to list the running EC2 instances under a given account and their region:

import boto3

# Replace with your own access key and secret
access_key = AKIA52GPOBQCO4SRMOWK
secret_key = txaF/m7lZnGQpppqCV4rCcsCvBHtlgtyE1BKBopb

# Create session
session = boto3.Session(
    aws_access_key_id=access_key,
    aws_secret_access_key=secret_key
)

# Get all regions
client = session.client('ec2')
regions = [region['RegionName'] for region in client.describe_regions()['Regions']]

# Iterate over each region
for region in regions:
    # Connect to EC2 in this region
    ec2 = session.client('ec2', region_name=region)
    # Get all running instances
    instances = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    # Print the instances
    for reservation in instances[Reservations]:
        for instance in reservation[Instances]:
            print(f'Instance {instance[InstanceId]} is running in {region}')

It also leaks the access_key and secret_key for the administration of the AWS account (Task 2).

Bucket Access

I’ll add “ec2.py” as a filter in my search and it returns 32 events:

Looking at the eventName for these events shows what’s happening:

Time (UTC)	Description	Reference
2023-01-24 22:36:58	`ec2.py` downloaded by attacker IP	CloudTrail [`GetObject`]
2023-01-24 22:37:59	`ec2.py` downloaded again by attacker IP	CloudTrail [`GetObject`]
2023-01-24 22:48:34	Initial log from attacker IP range	CloudTrail
2023-01-24 22:48:34	Create key pair `1337.key`	CloudTrail [`CreateKeyPair`]
2023-01-24 22:54:45 - 22:59:58	Start 6 instances of t2.micro EC2 (3 commands) with `1337.key`	CloudTrail [`RunInstances`]
2023-01-24 23:10:27 - 23:22:25	Enumerating / modifying security group rules [17 events]	CloudTrail [`ModifySecurityGroupRules` / `DescribeSecurityGroupRules`]
2023-01-24 23:25:55	Admin terminates 6 EC2 instances	CloudTrail [`TerminateInstances`]
2023-01-25 11:38:44	Created 1337 security group	CloudTrail [`CreateSecurityGroup`]
2023-01-25 11:51:02	Create 5 instances of t2.micro EC2 (1 command) with `1337.key`	CloudTrail [`RunInstances`]
2023-01-25 11:56:46 - 11:59:13	Enumerating / modifying security group ingress [6 events]	CloudTrail [`DescribeSecurityGroupRules` / `AuthorizeSecurityGroupIngress`]
2023-01-25 12:02:25	Create 1 instance of t2.micro EC2 (1 command) with `13337`	CloudTrail [`RunInstances`]
2023-01-25 13:41:44 - 13:57:34	Enumerating / modifying security group ingress [2 events]	CloudTrail [`AuthorizeSecurityGroupIngress`]
2023-01-25 13:59:00-14:03:02	Two failed instances and a successful instance of t2.micro EC2 with `13337`	CloudTrail [`RunInstances`]
2023-01-25 14:03	Last boot of PoshC2 C2 EC2 instance	CatScale [`last-utmp.txt`]
2023-01-25 14:35:02	Enumerating / modifying security group ingress [1 event]	CloudTrail [`AuthorizeSecurityGroupIngress`]
2023-01-25 14:35:02	Last activity from attack IP range	CloudTrail
2023-01-25 15:00:46 - 15:09:52	DESKTOP-R4KM0GJ\Marcus Athony victim interacts with PoshC2	PoshC2 logs
2023-01-25 15:11:37	ADmin terminates 6 EC2 instances	CloudTrail [`TerminateInstances`]

Challenge Info

Background

Scenario

Interview

Data

Artifact Background

Cat-Scale

CloudTrail

Tools

AWS Account Compromise

S3 Bucket Triage

ec2.py

Script

Bucket Access

Initial Access

AWS Activity

EventName Overview

Security Groups

Security Group Rules / Ingress

KeyPair Creation

Instance Creation

Instance Termination

VM Activity

CatScale

File Analysis

Overview

Database

Log

Results

Timeline

Question Answers