The Human-Friendly Guide: Incident Response & Threat Hunting in Microsoft Azure, Part 1

By Alon Klayman, Threat Hunting Expert at Team Axon

Series Introduction 

This is the first in a series of multiple blog posts, in which we’ll cover incident response and threat hunting in Microsoft Azure from the ground up.

Throughout this series, we have a few main goals we want to achieve:

• Systematically explain Azure components and their relevant attack surface
• Highlight key aspects and areas to look at when conducting IR or Threat Hunting against Azure infrastructure, specifying interesting log sources, as well as specific log records and fields that can be used as part of investigations
• Use practical examples, specifying different attack types and explaining how they can be identified
• Explain all of the covered topics in a human-friendly language. We want to avoid using tons of complicated terms and hard to understand jargon. The motto of this series is “A good explanation is a simple explanation.”

What's included in Part 1:

• Azure Attack Statistics and Popularity 
• Relationship Between Azure, Azure AD, and Microsoft 365
• Permissions in Azure/Azure AD
• Main Available Log Sources
• Log Fields to Focus On
• Example of an Azure incident and short investigation flow

Azure Attack Statistics and Popularity

The traditional computing world is being heavily transitioned towards the cloud, so of course, threat actors are following suit. Per one of Check Point’s published research pieces, there was a growth of 48% in cloud-based network attacks in 2022 compared to 2021. 

Azure Cloud, as one of the most popular cloud computing infrastructures, is one of the main targets associated with the massive growth in the number of cyber attacks. 

While the goals of the malicious actors remain pretty similar to the goals they had when attacking native on-premises infrastructure, the techniques they use when attacking cloud infrastructure are extremely different, which means the detection and investigations of those attacks are just as different as well.

It is also important to mention that cloud computing infrastructures and on-prem infrastructures sometimes have a connection between them, and this is especially common when talking about Azure use cases. We will also cover these hybrid use cases within this series.

Relationship Between Azure, Azure AD, and Microsoft 365

When talking about Azure, one of the most confusing things is the relationship between Azure, Azure AD, and Microsoft 365.

To keep it short, here is a simple description of each and the relationship between them:

• Azure (Azure Cloud): This is the cloud platform as a whole, including all of the different services and features that it provides. For example: compute services (VMs, Azure functions, etc.), networking services, storage, logic apps, and Azure Active Directory.
  • Azure Account: We can look at this one as the highest entity in the hierarchy, it provides access to Azure Services as a whole, and includes the Azure Subscriptions.
  • Azure Subscription:
    • An agreement with Microsoft to use one or more Microsoft cloud platforms or services. There are different types of subscriptions (agreements), including Microsoft 365 Subscriptions, Dynamics 365 subscriptions and Azure Subscriptions (Microsoft)
    • Another way to look at it is as some kind of “container” that includes different types of resources (for example Virtual machines, key vaults, etc.)
    • The Subscription is “attached” to an Azure AD organization/tenant (trust relationship)
    • The monthly billing is attached to a subscription (not to the Account as a whole)
    • You can have multiple subscriptions under the same account
• Azure Active Directory (Azure AD): As mentioned above, this is a service provided under the bigger umbrella of services of Azure Cloud. Azure AD is the cloud-based identity and access management service - or in simple terms - this is the main service that is responsible for authentication and authorization of users to Azure services, Azure portal, M365, SaaS applications, organizational internal resources, and any other service that supports Azure AD as an identity provider.
  • It is important to understand that an Azure AD tenant is automatically created whenever you use M365 or Azure. It is also important to know that there are Azure services that do not support Azure AD based authentication. Here is a list of services that support Azure AD Authentication.

• Microsoft 365: Previously known as Office 365, these are a set of services that are provided by Microsoft, including popular applications like Excel, Word, Teams, and OneDrive, as well as services like Exchange Online and more. 

The following diagram gives a high-level view of the relationship between the 3 main key terms mentioned above:

Diagram A: Azure, Azure AD, Microsoft365 - all connected

As can be seen in the diagram, Azure AD plays the role of gatekeeper, and is responsible for the access to both M365 services/applications and in part to Azure Services as well. So for example, if “Organization A” uses Azure as cloud infrastructure, as well as M365 services (including Exchange Online), and an employee tries to access the M365 panel, OWA, or Azure portal to manage Azure resources, he’ll need to “go through” and authenticate against Azure AD.

A key note to be aware of here is the fact that if your organization (or the organization you conduct an investigation for) uses M365 as part of its organizational infrastructure (e.g. the email infrastructure is Exchange Online), it means that this organization has its own Azure AD tenant. An Azure AD tenant will always exist whenever your organization uses M365 or Azure. This often comes as a surprise to many organizations that may not even be aware that they utilize an Azure AD tenant.

Permissions in Azure/Azure AD

Ok, so what about the permissions related to Azure and Azure AD? 

Understanding the different permission/role types exist in Azure/Azure AD is super important for understanding different attack types. Knowing which actions can be conducted by each permission type helps focus on the right areas and ask the right questions when conducting an investigation.

To keep it simple, you can think of it this way - there are 2 main sets of permissions in Azure Cloud: Azure roles (aka RBAC) and Azure AD roles.

• Azure Roles (aka RBAC): These roles include sets of permissions to conduct different activities against Azure resources. The easiest way to think of it is using Diagram A above. Azure Roles will allow the user who has this role assigned to them to conduct activities against the services on the right side of the diagram. For example, a user account with the Contributor role on a specific resource group, will have permissions to conduct broad set of activities against resources residing in Azure resource group, for example virtual machine. The following are the “main” built-in roles that are the most important to be familiar with:
  • Owner: Full rights to change the relevant resource and change the access control of it
  • Contributor: Very similar to Owner, without the option to change the access control of the resource
  • Reader: Read-only access to the relevant resource
  • User Access Administrator: Allows to manage the access control of a resource, but not to conduct changes on the resource itself

Example of Azure (RBAC) Roles

• Azure AD Roles: Azure AD has its own set of roles. This means that these permission sets are related to Azure AD managed parts, such as App Registrations, Enterprise Applications, Users, Service Principals, licenses, domain names, etc. Hence, if a user has permissions at the Azure AD level, it means she will have the ability to manage different aspects related to the Azure AD objects (e.g. users , groups, applications, etc), but NOT necessarily to Azure Resources.

• • Here are some examples of the must-know Azure AD Roles:
    • Global Administrator: This role provides the highest level of permissions in Azure AD. It allows the user to manage all the aspects of Azure AD, including conducting sensitive activities such as resetting the passwords of other users, adding and removing roles of other users, managing applications, etc. It's also important to mention that Global Administrators have access to services that use Azure AD identities, like Exchange Online, SharePoint Online, Microsoft Purview Compliance Portal, etc. 
    • Application Administrator: Can manage all the aspects of App Registrations and Enterprise Applications
    • Helpdesk Administrator: Can reset the passwords of non-administrators user accounts
    • Exchange Administrator: can manage all aspects of Exchange Online
    • See here for more details about the specific must-know Azure AD Roles and other Azure AD Roles 

Table 1: Main Azure and Azure AD Roles

But wait Exchange Administrator is capable of managing Exchange Online, which is a M365 Service/Product that is not part of Azure AD. Why is it mentioned under Azure AD Roles?

This is because, as mentioned before, Azure AD plays a big part in authentication and authorization for different services, including M365. Exchange Administrator (as well as other roles) can be managed in both Azure AD Admin Center and by M365 Admin Portal. Assigning a role to a user in one of them, will make it effective both for Azure AD and M365.

OK - it’s starting to make sense now. But, if a user has a Global Administrator Role, it means she can do everything in the Azure AD Tenant and Azure Subscription, right?

The answer here is a bit complicated, but we’ll try to simplify it as much as possible. The very short answer is: No. This is because Azure AD role assignments do not grant access to Azure Subscription resources, and vice versa.

But, Global Administrators specifically have the ability to “Elevate” their access. This means that the Global Administrator can provide himself an Azure role of “User Access Administrator” to the “root” scope, which means that the Global Administrator will be able to manage the access control of all the subscriptions (and resources under them) in the Azure AD Tenant/directory.

Available Log Sources

Now, after this quick overview of the basics of Azure Cloud as it relates to IR and Threat Hunting, let’s talk business. And by business, I mean logs.

Azure Cloud includes many different log sources, but let’s first focus on the 3 main ones:

• Azure Sign-in Logs: These logs provide information about sign-ins to Azure AD, including different types of sign-ins (interactive, non-interactive, Service principal sign-ins etc.)
• Azure Audit Logs: These logs include information about events in Azure AD, for example, changes to applications, groups, users, etc.
• Azure Activity Logs: Azure Activity logs provide information about events which took place on the subscription-level in Azure. (for example, creation of VM, modification of a VM, creation of a key vault, etc.)

The 3 log sources mentioned here are the ones you as incident responder/threat hunter/security analyst/any other security title of your choice, have to be familiar with when dealing with Azure-related investigations. 

Between these 3 log sources, there are 2 that tend to cause confusion: Audit logs and Activity logs. 

Let’s make it simple to distinguish between these two, using our lovely diagram:

Diagram B: Azure, Azure AD Log Sources

The general rule of thumb, as can be seen in the diagram, is as follows:

• Azure Activity logs are related to the Azure subscription level (Resources, etc.).
• Azure AD Audit logs are related to Azure AD tenants, actions conducted against Azure AD users, groups, app registrations, enterprise applications, etc.

Remembering this makes it much easier to understand when is the right time to look at each of the log sources.

So that’s it? There are only three log sources?
Of course not. These are the main three log sources to be aware of when investigating Azure incidents, but there are many more including Azure key vault logs, Azure VM-related logging, Azure Storage logs, etc.

These additional logs can provide more information about events conducted against the organizational Azure resources. For example, if we search Azure Activity logs for logged activities related to Azure Key Vault (access to Azure Key Vault or reading secrets), we won’t be able to get comprehensive information directly from these logs. You can think of those additional log sources as a kind of “improvements” to the 3 main log sources, and their importance can be crucial for some investigations.

Note: Those additional log sources (in contrast to the three main ones) won’t always be enabled by default, and should be explicitly pre-configure.

• Category: You can choose between multiple Sign-in categories in the Sign-in logs page. Today there are 4 category types: User Sign-ins, Non-Interactive User Sign-ins, Managed Identities Sign-ins, and Service Principal Sign-ins. Looking at the correct category is crucial to make sure you are not missing sign-in events.
• Location Tab
  • IP address: The source IP from which the Sign-in originated from.
  • Location: geographical location identified as related to the sign-in attempt.
• Basic Info Tab
  • Authentication requirements: provides an indication about the authentication requirements needed for authentication (single-factor authentication, multifactor authentication, etc.)
  • Status: Was the sign-in successful? 
  • User & Username: The identity of the signed-in entity 
  • User Type: the type of the signed-in user account (member, guest, etc.)
  • Client app: provides an indication about the authentication clients that had been used to sign-in (e.g. browser, mobile apps and desktop clients, etc.).
  • Application: Can provide indication about the target application/service to which the user had been authenticated (Azure Portal, Microsoft Azure CLI, Microsoft Azure PowerShell, OfficeHome, Microsoft Office 365 Portal, etc.)
  • User Agent: The user-agent identified as being used as part of the sign-in attempt.

• Resource: the relevant resource identified related to the Activity log entry. The resource identified is being presented in a URN format: /subscriptions/<subscription id>/resourceGroup/<resource group name>/providers/<resource provider namespace>/<resource type>/<resource name>
• Operation Name: Indicates the actual operation that occurred. For example: “Restart Virtual Machine”, “Create or Update Virtual Machine Extension”
• Event initiated by: the identifier (UPN) of the entity that conducted the action.
• JSON Tab:  includes a JSON view, which includes in-depth information about the log entry, including some of the information mentioned above, as well as other useful fields like:
  • action → the exact action using URN identification. For example, for operation name of “Validate Deployment” we’ll see action value of: “Microsoft.Resources/deployments/validate/action”
  • ipaddr → Can provide an indication about the source IP address from which the logged action had been conducted.
  • name → The name of the identity conducted the logged action.

Example of Azure Cyber Incident and Investigation Steps

Now that we have a better understanding of the log sources and when we can use each of them, let’s look at a practical example, in which you’ll play the role of an IR team member.

In this example, we’ll demonstrate how each of the three main log sources can be used to investigate an incident.

Incident Background Story

Let’s pretend that you are in the middle of an incident response investigation related to an attack conducted against “HNTR-Global”, a (fake) shipping company located in the United States. The CISO of HNTR-Global reached out to your company on August 27, 2023, asking for incident response services. He informed you and your team about the current situation:

“We got an indication that some sensitive information related to HNTR-Global had been offered for sale over the dark web a few days ago. The leaked information is located on a specific application server (Server Name: HNTR-AppSERV-01 ; OS: Windows Server 2019) that we have in our organizational Azure infrastructure, in the hntrsch.onmicrosoft.com domain. This leads us assume that there was unauthorized access to this server in order to steal the information.

There weren’t any default credentials that could have been used to access this server, and it was fully patched. We want to contain the incident ASAP, and to simultaneously investigate it to get a better understanding of the scope of the incident and the way this unauthorized access had been achieved.” 

Phase A (Activity logs): 

One of your colleagues, an IR team member, started to look at the logs of the application server itself (Windows Event logs and applications logs in this case), trying to get an indication about how it had been accessed.

You, as another IR Team member, have the responsibility to investigate the incident focusing on the relevant Azure infrastructure. 

The first thing you do, is look at the Azure Activity logs related to the compromised server (HNTR-AppSERV-01).

Screenshot 2: HNTR-AppSERV-01 Azure Activity Logs

As can be seen in the screenshot above, you identify multiple Activity log entries related to this virtual machine. Those log entries describe actions conducted by “DevOps-Azr-01@hntrsch.onmicrosoft[.]com”.

Most of them are related to “Create or Update Virtual Machine Extension” event type. Azure Virtual Machine extensions can be thought of as applications that can be used for post-deployment configuration and automation on Azure VMs (for example: software installation, execution of script inside a VM, and more).

Here is a list of common VM extension types:

Screenshot 3: Common VM Extension (Microsoft Reference)

By looking at the type of the VM extension related to HNTR-AppSERV-01 exists in the logs, you identify the type of extension used there was “enablevmaccess” extension:

Screenshot 4: Azure Activity Log - Update of Virtual Machine Extension

The “enablevmaccess” extension allows to reset the login credentials of local users of the virtual machine. This extension is intended to be used when an authorized user forgot or lost the credentials to access the virtual machine. However, it can be abused by threat actors as well.
The user initiated the “Create or Update Virtual Machine Extension” action was “DevOps-Azr-01@hntrsch.onmicrosoft[.]com”.

Now that you have an initial understanding of the artifacts in the investigated Azure infrastructure you sync with the other member of the IR investigation team who investigates the application server. You, who looked at the artifacts from the Azure infrastructure perspective, informed her about the installation of the extension intended for credentials reset, asking her to verify if they also see the indication of it using the relevant log file on the VM (C:\WindowsAzure\Logs\WaAppAgent.log). She claimed that indeed they identified an indication of “enablevmaccess” installation in this log as well.

In addition, you specify the timestamp of the extension related activity logs (17:53-17:54 GMT +0300) asking her to look at creations of new users at this time, as well as suspicious logon events around this time.

After some checks conducted by her, she updates you about multiple interesting findings from the windows event logs exist on the application server:

• Security event ID 4720 at 17:54 indicates of a creation of user named “administrat0r”.
• Security event ID 4732 at 17:54 indicates of the addition of “administrat0r” to “Administrators” local group.
• Successful RDP connection at 18:01 towards the server using a local user account named “administrat0r” user account toward “HNTR-AppSERV-01” originated from a non-organizational external IP address.

She also mentioned that there were only 3 RDP connections toward this server over the last 3 months. The other two were about two months ago from a known organizational IP address, which makes the specific RDP connection mentioned above highly suspicious to be related to the exfiltration.

Diagram C: Incident Investigation - Checkpoint A

Another key insight was the fact that the creation of the extension had been conducted shortly before the creation of a new local user account, the addition of it to the local administrators group, and a suspicious RDP connection to the application server we got informed about by collaborating with other IR team member that conducted the host-level investigation.

Phase B (Audit logs):

Quickly checking with the HNTR-Global IT and DevOps teams, you understand that this DevOps-Azure-01 user account is not being regularly used, and it has been created as a backup user account. The teams were surprised by the usage of it to reset the password of the relevant VM.

Looking at the Azure Sign-in Logs related to this user account, you see that there was only one sign-in event related to this user account from the last 6 months, and it was shortly before the password reset we identified.

The authentication type was Single-Factor authentication, meaning no MFA had been configured for this user account.

The DevOps team leader you talk with claims that the password of this user account was a super complex one, including more than 20 characters, so an unauthorized access to this user account using some kind of password guessing or brute force attack is very unlikely. 

Looking at the Azure Audit Logs related to this DevOps user account, you found another key insight: a password reset had been conducted against this user account, by the following user: Santon.James@hntrsch.onmicrosoft.com, on August 23, 2023.

Screenshot 5: Azure Audit Log - Password Reset by Administrative user

The password reset, as can be seen above, took place at 16:58, which is about an hour before the ‘enablevmaccess’ usage we identified using the Azure Activity logs.

The log entry provides information about the logged activity itself:

Screenshot 6: Azure Audit Log - Password Reset by Administrative user - Activity Info

Note: The IP address field in Audit log entries might appear as Microsoft IP Address. This is the case in the Audit log entry above. The IP mentioned there is not the actual IP address from which the user conducted the action of password reset. This means we don’t get an indication about the potential source IP address of the threat actor from this log entry in this case. This is important to pay attention to during investigations, to avoid using a benign Microsoft IP address as an IOC.

The Target tab provides an indication about the target object of which password had been reset. Here we can see both the UPN and the Display Name. The display name provides us with some information about this user account, which in this case is: “DevOps Operations Backup User.” This aligns with the information from the DevOps team leader about the nature of this user.

Screenshot 7: Azure Audit Log - Password Reset by Administrative user - Target(s) 

From a quick check, James Santon is the Help-Desk team leader, but from quick questioning, you understand that he is not aware of the password reset conducted to the DevOps user account. 

Note: James, as mentioned above is the Help-Desk team leader. From looking at the Azure AD Roles assigned to him, it can be quickly identified that he is indeed a “Helpdesk Administrator.” This is the reason that he could reset the password of another Azure AD user account.

Looking at the audit logs of actions initiated by Santon.James@hntrsch[.]onmicrosoft[.]com and filtering the audit logs specifically by its user account, you manage to find no indication of other events, aside from the password reset of “DevOps-Azr-01@hntrsch[.]onmicrosoft[.]com,” had been conducted by this user account.

Diagram D: Incident Investigation - Checkpoint B

Phase C (Sign-in logs) - Successful Brute-Force attack

So after getting the indication about the 2 potentially compromised user accounts, it is important to check where their sign-ins had been conducted from (IP and location), and what we can learn from the sign-in logs we identify.

To do so, you look at the Azure Sign-in logs. 

First, as mentioned above, you looked at the Sign-in logs of the DevOps user account and could clearly see that there were no sign-in logs using this user account for about 6 months prior to August 23, 2023. 

The IP from which the Sign-in to this user account had been conducted wasn’t a known organizational IP (62.x.x.x), and it was geographically associated with Tel-Aviv (no known organizational activity was conducted from this geographical location before).

By looking at the Sign-in logs of the second user account you quickly identify that sign-in attempts from the same IP address used to sign-in to the DevOps user account had also occurred. 

This means that it is safe to move forward with the assumption that we are looking at an IP address that was associated with the threat actor that gained unauthorized access to HNTR-Global’s Azure infrastructure.

Using this IP address as a pivot artifact, you look for Sign-in attempts from this IP address over the last 3 months. Here, you identify what seems to be a low-rate brute-force attack against 3 organizational Azure AD user accounts: Roman Paul, James Grant, and James Santon.
All of the sign-in attempts were conducted on August 23, 2023.

As can be seen in the following screenshots, there were multiple brute-force “waves” targeting one user at a time. 

Here are some relevant Sign-in log entries as they look like in the Azure portal:

Screenshot 8: Azure Sign-in logs - Failed Login Attempts 23.08.23 - James Grant & Roman Paul

Screenshot 9: Azure Sign-in logs - Successful Slow-rhythm Brute-Force - 23.08.23 - James Santon

Luckily, after looking at the logs, it appears that no successful sign-in attempts were conducted using James Grant and Roman Paul’s user accounts. However, as can be seen above, a successful sign-in attempt using James Santon user account had been identified on August 23, 2023 between 04:48 to 04:57 PM (GMT +0300).

Below is an example of a successful sign-in log entry. We get a lot of information including:

• The sign-in was “Single-factor authentication,”
• The application that was used/accessed by the threat actor was “Azure Portal,”
• The sign-in was conducted by using a “Browser” client app
• The exact user agent that had been used

Screenshot 10: Azure Sign-in logs - Successful Unauthorized Sign-in Log  - 23.08.23 - James Santon

Ok - that was a lot. Let’s take a breath and summarize the investigation.

ATTACK FLOW SUMMARY:

Diagram D: Full Incident Investigation - Summary

The attack flow we investigated was a relatively simple one, and demonstrated how the different main Azure log sources can be used together for investigation purposes.

In addition, we also demonstrated the power of correlating the native Azure logs with other log sources, such as Host-level artifacts, to get a more comprehensive understanding of the investigated attack.

The Human-Friendly Guide - Future Plans

As we continue into future parts of this series, we will continue to expand our knowledge of incident response and threat hunting in Microsoft Azure. We will continue to learn additional terms and concepts like Azure AD Enterprise Applications, App registrations, touch on subjects related to M365, and more.

We will also dive into more examples of investigations, covering additional attack types, using even more types of Azure logs. 

Until then, use this as a reference for you and your team as you navigate the Microsoft Azure landscape. Be sure to stay tuned and happy hunting!

To stay updated on threat hunting research, activities, and queries, follow Team Axon’s Twitter account (@team__axon).