Microsoft Sentinel in Action: Architect, design, implement, and operate Microsoft Sentinel as the core of your security solutions

BIRMINGHAM—MUMBAI

Microsoft Sentinel in Action

Second Edition

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Vijin Boricha

Publishing Product Manager: Meeta Rajani

Senior Editor: Arun Nadar

Content Development Editor: Sulagna Mohanty

Technical Editor: Arjun Varma

Copy Editor: Safis Editing

Project Coordinator: Shagun Saini

Proofreader: Safis Editing

Indexer: Vinayak Purushotham

Production Designer: Vijay Kamble

First published: May 2020

Second edition: January 2022

Production reference: 1021221

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80181-553-6

www.packt.com

Contributors

About the authors

Richard Diver is a senior technical business strategy manager for the Microsoft Security Solutions group, focused on developing security partners. Based in Chicago, Richard works with advanced security and compliance partners to help them build solutions across the entire Microsoft platform, including Microsoft Sentinel, Microsoft Defender, Microsoft 365 security solutions, and many more. Prior to Microsoft, Richard worked in multiple industries and for several Microsoft partners to architect and implement cloud security solutions for a wide variety of customers around the world. Any spare time he gets is usually spent with his family.

Gary Bushey is an Azure security expert with over 25 years of IT experience. He got his start early on when he helped his fifth-grade math teacher with their programming homework and worked all one summer to be able to afford his first computer, a Commodore 64. When he sold his first program, an apartment management system, at 14 he was hooked. During his career, he has worked as a developer, consultant, trainer, and architect. When not spending time in front of a computer, you can find him hiking in the woods, taking pictures, or just picking a direction and finding out what is around the next corner.

John Perkins is the founder and principal of Threat Angler, a cybersecurity service provider that specializes in managed services, professional services, and training with a focus on delivering cybersecurity outcomes to customers of all shapes and sizes. John has over 20 years of experience in cybersecurity and has contributed to nearly all cybersecurity disciplines during his career. He has experience with numerous applications, including Microsoft Sentinel, and has designed, built, and led managed security services for several large service providers. In his free time, John enjoys spending time with his family, traveling, and staying active.

About the reviewers

Ashwin Patil currently works as a senior program manager for Microsoft Threat Intelligence Center (MSTIC) and has over 10 years' experience entirely focused on security monitoring and incident response, defending enterprise networks. In his current role, he primarily works on threat hunting, detection research in Kusto Query Language (KQL) for Microsoft Sentinel, and developing Jupyter notebooks written in Python/R to do threat hunting and investigation across a variety of cloud and on-premises security event log data sources. He has a bachelor's degree in computer engineering and possesses various SANS certifications, including GCIA, GCFE, and GCIH in the field of Digital Forensics and Incident Response (DFIR).

Dennis Pike is the original sales engineer at Island, a stealth mode security startup. He would tell you more, but they may put him on an island with nothing but a volleyball. Born in Kentucky, he surprisingly can't stand bourbon and ended up a nationally ranked beer judge instead. He holds a BSc in systems engineering from the University of Virginia and has spent the last 25 years working in IT, including as a Global Black Belt – Advanced Security Analytics at Microsoft where he focused on Microsoft Sentinel.

I want to thank my wife, Heather, for her patience, love, and support.

Rod Trent is a senior cloud security advocate for Microsoft and an Microsoft Sentinel global SME helping customers migrate from existing SIEMs to Microsoft Sentinel to achieve the promise of better security through improved efficiency without compromise. He is a husband, dad, and first-time grandfather (so speak slowly and loudly). He spends his spare time (if such a thing does truly exist) simultaneously watching Six Million Dollar Man episodes and writing KQL queries.

Table of Contents

Preface

Section 1: Design and Implementation

Chapter 1: Getting Started with Microsoft Sentinel

The current cloud security landscape

The cloud security reference framework

SOC platform components

Mapping the SOC architecture

Log management and data sources

Operations platforms

Threat intelligence and threat hunting

SOC mapping summary

Security solution integrations

Cloud platform integrations

Integrating with Amazon Web Services (AWS)

Integrating with Google Cloud Platform (GCP)

Integrating with Microsoft Azure

Private infrastructure integrations

Service pricing for Microsoft Sentinel

Scenario mapping

Step 1 – defining the new scenarios

Step 2 – explaining the purpose

Step 3 – the kill chain stage

Step 4 – which solution will perform detection?

Step 5 – what actions will occur instantly?

Step 6 – severity and output

Step 7 – what action should the analyst take?

Summary

Questions

Further reading

Chapter 2: Azure Monitor – Introduction to Log Analytics

Technical requirements

Introduction to Azure Monitor Log Analytics

Planning a workspace

Creating a workspace using the portal

Creating a workspace using PowerShell or the CLI

Creating an Azure Resource Management template

Using PowerShell

Using the CLI

Exploring the Overview page

Managing permissions for the workspace

Enabling Microsoft Sentinel

Exploring the Microsoft Sentinel Overview page

The header bar

The summary bar

The Events and alerts over time section

The Recent incidents section

The Data source anomalies section

The Potential malicious events section

The Democratize ML for your SecOps section

Connecting your first data source

Obtaining information from Azure virtual machines

Advanced settings for Log Analytics

Agents management

The Agents configuration options

Computer Groups

Summary

Questions

Further reading

Section 2: Data Connectors, Management, and Queries

Chapter 3: Managing and Collecting Data

Choosing data that matters

Understanding connectors

Native connections – service to service

Direct connections – service to service

API connections

Agent-based

Configuring Microsoft Sentinel connectors

Configuring Log Analytics storage options

Calculating the cost of data ingestion and retention

Reviewing alternative storage options

Summary

Questions

Further reading

Chapter 4: Integrating Threat Intelligence with Microsoft Sentinel

Introduction to TI

Understanding STIX and TAXII

Choosing the right intel feeds for your needs

Implementing TI connectors

Enabling the data connector

Registering an app in Azure AD

Configuring the MineMeld TI feed

Confirming the data is being ingested for use by Microsoft Sentinel

Summary

Questions

Further reading

Chapter 5: Using the Kusto Query Language (KQL)

Running KQL queries

Introduction to KQL commands

Tabular operators

Query statements

The let statement

Scalar functions

The ago() function

String operators

Summary

Questions

Further reading

Chapter 6: Microsoft Sentinel Logs and Writing Queries

An introduction to the Microsoft Sentinel Logs page

Navigating through the Logs page

The page header

The Tables pane

The Queries pane

The Functions pane

The Filter pane

The KQL code window

Running a query

The Results window

Learn more

Writing a query

The billable data ingested

Map view of logins

Other useful tables

Summary

Questions

Further reading

Section 3: Security Threat Hunting

Chapter 7: Creating Analytic Rules

An introduction to Microsoft Sentinel Analytics

Types of analytic rules

Navigating through the Analytics home page

Creating an analytic rule

Creating a rule from a rule template

Creating a new rule using the wizard

Managing analytic rules

Summary

Questions

Further reading

Chapter 8: Creating and Using Workbooks

An overview of the Workbooks page

The workbook header

The Templates view

Workbook detail view

Missing required data types

Saved template buttons

Walking through an existing workbook

Creating workbooks

Creating a workbook using a template

Creating a new workbook from scratch

Editing a workbook

Advanced editing

Managing workbooks

Workbook step types

Text

Query

Metric

Parameters

Links/tabs

Groups

Advanced Settings

Style

Summary

Questions

Further reading

Chapter 9: Incident Management

Using the Microsoft Sentinel Incidents page

The header bar

The summary bar

The search and filtering section

Incident listing

Incident details pane

Using the Actions button

Exploring the full details page

The Timeline tab

The Alerts tab

The Bookmarks tab

The Entities tab

The Comments tab

Investigating an incident

Showing related alerts

The Timeline button

The Info button

The Entities button

The Insights button

The Help button

Summary

Questions

Further reading

Chapter 10: Configuring and Using Entity Behavior

Introduction to Microsoft Sentinel Entity behavior

Enabling Entity behavior

Overview of the Entity behavior page

The header bar

The search section

Entities with alerts

Overview of the Entity behavior details page

Identifying information

Notable events

Insights

Creating Entity behavior queries

Header bar

Activities list

Activity details pane

Adding a new activity

Summary

Questions

Further reading

Chapter 11: Threat Hunting in Microsoft Sentinel

Introducing the Microsoft Sentinel Hunting page

The header bar

The summary bar

The hunting queries list

Hunting query details pane

Working with Microsoft Sentinel hunting queries

Adding a new query

Editing a query

Cloning a query

Deleting a query

Adding to Livestream

Creating an analytics rule

Working with livestream

Working with bookmarks

Creating a bookmark

Viewing bookmarks

Associating a bookmark with an incident

Using Microsoft Sentinel notebooks

The header bar

The summary bar

The notebook list

The notebook details pane

Creating a workspace

Performing a hunt

Developing a premise

Determining data

Planning a hunt

Executing an investigation

Responding

Monitoring

Improving

Summary

Questions

Further reading

Section 4: Integration and Automation

Chapter 12: Creating Playbooks and Automation

Introduction to Microsoft Sentinel playbooks

Introduction to Microsoft Sentinel Automation

The header bar

The summary bar

Automation rules listing

Adding a new automation rule

Playbook pricing

Types of playbooks

Overview of the Microsoft Sentinel connector

Exploring the Playbooks tab

Logic app listing

Logic app settings page

The menu bar

The header bar

The essentials section

The Runs history section

Creating a new playbook

Using the Logic Apps Designer page

The Logic Apps Designer header bar

The Logic Apps Designer workflow editor section

Creating a simple Microsoft Sentinel playbook

Summary

Questions

Further reading

Chapter 13: ServiceNow Integration for Alert and Case Management

A brief history of Microsoft Sentinel and ServiceNow integration

Integrating Microsoft Sentinel with ServiceNow ITSM using Microsoft Sentinel Logic Apps

Integrating Azure security alert sources (not just Sentinel) with ServiceNow Security Incident Response via the Microsoft Graph Security API

Integrating Microsoft Sentinel with ServiceNow Security Incident Response via an API directly to Microsoft Sentinel

Steps to integrate Microsoft Sentinel with ServiceNow

Configuring the Microsoft Azure portal

Installing the Microsoft Sentinel integration plugin in ServiceNow

Configuring the ServiceNow Sentinel plugin to authenticate to Microsoft Sentinel

Creating profiles in the ServiceNow Sentinel integration plugin

Summary

Section 5: Operational Guidance

Chapter 14: Operational Tasks for Microsoft Sentinel

Dividing SOC duties

SOC engineers

SOC analysts

Operational tasks for SOC engineers

Daily tasks

Weekly tasks

Monthly tasks

Ad hoc tasks

Operational tasks for SOC analysts

Daily tasks

Weekly tasks

Monthly tasks

Ad hoc tasks

Summary

Questions

Chapter 15: Constant Learning and Community Contribution

Official resources from Microsoft

Official documentation

Tech community – blogs

Tech community – forums

Feature requests

LinkedIn groups

Other resources

Resources for SOC operations

MITRE ATT&CK® framework

National Institute of Standards for Technology (NIST)

Using GitHub

GitHub for Microsoft Sentinel

GitHub for community contribution

Specific components and supporting technologies

Kusto Query Language

Jupyter Notebook

Machine learning with Fusion

Azure Logic Apps

Summary

Assessments

Other Books You May Enjoy

Preface

Microsoft Sentinel is an intelligent security service developed by Microsoft with a focus on integrating and bringing together cloud security and artificial intelligence. Microsoft Sentinel in Action will help you to gain enough understanding to make the most of Azure services to secure your environment against modern cybersecurity threats.

During Ignite 2021, Microsoft announced that Azure Sentinel will be renamed to Microsoft Sentinel. However, changing the name everywhere is a very time consuming task and, as of when this book was finished, the process was not yet completed. Due to this, some images shown still show Azure Sentinel rather than Microsoft Sentinel as the name change were yet to be completed in the Azure portal, however, the functionality is still the same.

Who this book is for

If you are an IT professional with prior experience in other Microsoft security products and Azure and are now looking to expand your knowledge to incorporate Microsoft Sentinel, then this book is for you. Security experts using an alternative SIEM tool who want to adopt Microsoft Sentinel as an additional service or as a replacement will also find this book useful.

What this book covers

Chapter 1, Getting Started with Microsoft Sentinel, includes an overview of the cloud security architecture as we see it today. This information lays a foundation for understanding what a modern Security Operations Center (SOC) is and why Microsoft Sentinel plays a key role. We will focus on the specific components needed to create a SOC platform and how Microsoft Sentinel brings all the data together to provide a central analysis and action capability.

Chapter 2, Azure Monitor – Introduction to Log Analytics , focuses on the creation of the Azure Log Analytics workspace, which is where we store all the log data for Microsoft Sentinel to analyze. This is an important first step in configuring Microsoft Sentinel.

Chapter 3, Managing and Collecting Data, teaches you how to collect data, manage the data to prevent overspend, and query the data for useful information as part of your threat hunting and other security activities.

Chapter 4, Integrating Threat Intelligence with Microsoft Sentinel, explains the options available for adding threat intelligence feeds to Microsoft Sentinel to enable the security team to have a greater understanding of the potential threats against their environment. Threat intelligence feeds add contextual information to the data gathered from logs across the organization.

Chapter 5, Using the Kusto Query Language, provides an introduction to the Kusto Query Language (KQL) and has some sample queries for you to work out for yourself.

Chapter 6, Microsoft Sentinel Logs and Writing Queries, expands on the skills learned in Chapter 5, Using the Kusto Query Language, to create useful Microsoft Sentinel queries to discover anomalous behaviors and patterns of activity.

Chapter 7, Creating Analytic Rules, teaches you how to take KQL queries and use them to create Microsoft Sentinel analytic rules to create incidents.

Chapter 8, Creating and Using Workbooks, explains the concept of workbooks, how to use workbook templates, how to edit an existing workbook, and how to create your own workbook.

Chapter 9, Incident Management, discusses Microsoft Sentinel incidents, what they are, how to manage them, and how to investigate them.

Chapter 10, Configuring and Using Entity Behavior, teaches you another way of obtaining more information about your incident by using Entity behavior.

Chapter 11, Threat Hunting in Microsoft Sentinel, discusses Microsoft Sentinel hunting queries and how to use them and touches upon Jupyter notebooks.

Chapter 12, Creating Playbooks and Automation, provides an overview of Microsoft Sentinel playbooks and will discuss using the Microsoft Sentinel trigger and actions to perform automations.

Chapter 13, ServiceNow Integration for Alerts and Case Management, expands upon what was learned in Chapter 12, Creating Playbooks and Automation, to provide a step-by-step guide on how to create a workflow that creates a ServiceNow ticket from an Microsoft Sentinel alert. These same steps could be modified to work with any ticketing agent.

Chapter 14, Operational Tasks for Microsoft Sentinel, will provide some operational guidance on various tasks that should be performed daily, weekly, monthly, and as needed.

Chapter 15, Constant Learning and Community Contribution, will finish the book by offering guidance on where to get the latest information and how to contribute to the community that is growing to support the development and sharing of security-related information and techniques.

To get the most out of this book

We recommend that you have access to an Azure environment where you have the proper rights to create your Micosoft Sentinel environment. Prior usage of the Azure portal would also be beneficial.

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781801815536_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: In the following example, when looking at the rows where the state is NORTH CAROLINA, all the columns other than State and duration will be empty since the NCEvents table only has the State and duration columns.

A block of code is set as follows:

let FLEvents = StormEvents

| where State == FLORIDA;

let NCEvents = StormEvents

| where State == NORTH CAROLINA

| project State, duration = EndTime - StartTime;

NCEvents | union FLEvents

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: To run the samples for this chapter, you will need to expand the Samples logs on the left-hand side of the screen and then select StormEvents.

Any command line input or output is written as follows:

StormEvents

| distinct State

| order by State asc

Tips or Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at customercare@packtpub.com.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packt.com with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you've read Microsoft Sentinel in Action, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.

Section 1: Design and Implementation

In this section, you will get an overview of Microsoft Sentinel, including the current cloud landscape, the cloud security reference framework, Security Operations Center (SOC) platform components, and how to map the architecture. You will also learn about the Azure Monitor Log Analytics resource, including planning your Log Analytics instance, how to create a new instance, and attaching it to Microsoft Sentinel.

This section contains the following chapters:

Chapter 1, Getting Started with Microsoft Sentinel

Chapter 2, Azure Monitor – Introduction to Log Analytics

Chapter 1: Getting Started with Microsoft Sentinel

Welcome to the first chapter in this book about Microsoft Sentinel. To understand why this solution was developed and how best to use it in your organization, we need to explore the cloud security landscape and understand each of the components that may feed data into, or extract insights from, this system. We also need to gain a baseline understanding of what a strong Security Operations Center (SOC) architecture looks like, and how Microsoft Sentinel is going to help build the foundations for a cost-effective and highly automated cloud security platform.

In this chapter, we will cover the following topics:

The current cloud security landscape

The cloud security reference framework

SOC platform components

Mapping the SOC architecture

Security solution integrations

Cloud platform integrations

Private infrastructure integrations

Service pricing for Microsoft Sentinel

Scenario mapping

The current cloud security landscape

To understand your security architecture requirements, you must first ensure that you have a solid understanding of the IT environment that you are trying to protect. Before deploying any new security solution, there is a need to map out the solutions that are currently deployed and how they protect each area of the IT environment. The following list provides the major components of any modern IT environment:

End user habits that are counter-productive to security endeavors

Identity for the authentication and authorization of access to systems

Networks to gain access to internal resources and the internet

Storage and compute in the data center for internal applications and sensitive information

End user devices and the applications they use to interact with data

And in some environments, you can include Industrial Control Systems (ICS) and the Internet of Things (IoT)

When we start to look at the threats and vulnerabilities for these components, we quickly find ourselves deep in the alphabet soup of problems and solutions.

Figure 1.1 – The alphabet soup of cybersecurity

This is by no means an exhaustive list of the potential acronyms available. Understanding these acronyms is the first hurdle; matching them to the appropriate solutions and ensuring they are well deployed is another challenge altogether (a table of these acronyms can be found in the appendix of this book).

The cloud security reference framework

To assist with the discovery and mapping of current security solutions, we developed the cloud security reference framework. The following diagram is a section of this framework that provides the technical mapping components, and you can use this to carry out a mapping of your own environment:

Figure 1.2 – Technical mapping components; the cloud security reference framework

Each of these 12 components is described in the following list, along with some examples of the types of solutions to consider as they relate to integration with Microsoft Sentinel and the rest of your security architecture:

Security Operations Center: At a high level, this includes the following technologies and procedures: log management and Security Incident and Event Monitoring (SIEM), Security Orchestration and Automated Response (SOAR), vulnerability management, threat intelligence, incident response, and intrusion prevention/detection. This component is explored further in the Mapping the SOC architecture section later in this chapter.

Productivity Services: This component covers any solution currently in use to protect the business productivity services that your end users rely on for their day-to-day work. This may include email protection, SharePoint Online, OneDrive for Business, Box, Dropbox, Google apps, and Salesforce. Many more will appear in the future, and most of these should be managed through a Cloud Access Security Broker (CASB) solution.

Identity and Access Management: Identities are among the most important entities to track. Once an attacker gains access to your environment, their main priority is to find the most sensitive accounts and use them to exploit systems further. In fact, identity is usually one of the first footholds in your IT environment, usually through a successful phishing attack. A simple resolution is to implement multi-factor authentication, ensuring that even if a password is stolen (or guessed), the attacker would need multiple attempts to access the system.

Client Endpoint Management: This component covers a wide range of endpoints, from desktops and laptops to mobile devices and kiosk systems, all of which should be protected by specialized solutions such as Endpoint Detection and Response (EDR), Mobile Device Management (MDM), and Mobile Application Management (MAM) solutions to ensure protection from advanced and persistent threats against the operating systems and applications. This component also includes secure printing, managing peripherals, and any other device that an end user may interact with, such as the future of virtual reality/augmentation devices.

Cloud Access Security Broker (CASB): This component has been around for several years and is finally becoming a mainstay of modern cloud security infrastructure due to the increased adoption of cloud services. The CASB is run as a cloud solution that can ingest log data from Software as a Service (SaaS) applications and firewalls and will apply its own threat detection and prevention solutions. Information coming from the CASB will be consumed by the SIEM solution to add to the overall picture of what is happening across your diverse IT environment.

Perimeter Network: One of the most advanced components, when it comes to cybersecurity, must be the perimeter network. This used to be the first line of defense and for some companies still is the only line of defense. That is changing now, and we need to be aware of the multitude of options available; from external-facing advanced firewalls, web proxy servers, and application gateways to virtual private networking solutions and secure DNS, this component will also include protection services such as Distributed Denial of Service (DDoS), Web Application Firewall (WAF), and intrusion protection/detection services.

IoT and Industrial Control Systems: ICS are usually operated and maintained in isolation from the corporate environment, known as the Information Technology/Operational Technology (IT/OT) divide. These are highly bespoke systems that may have existed for decades and are not easily updated or replaced. The networks and devices may be highly sensitive to any latency or attempts to scan; instead, the recommended approach is passive monitoring of network traffic.

The reference to IoT is different, yet similar; in these systems, there will be a lot of small devices that collect data and control critical business functions without working on the same network. Some of these devices can be smart to enable automation; others are single-use (vibration or temperature sensors). The volume and velocity of data that can be collected from these systems can be very high. If useful information can be gained from the data, then consider filtering the information before ingesting it into Microsoft Sentinel for analysis and short- or long-term retention.

Private Cloud Infrastructure: This may be hosted in local server rooms, a specially designed data center, or hosted with a third-party provider. The technologies involved in this component will include storage, networks, internal firewalls, and physical and virtual servers. The data center has been the mainstay of many companies for the last 2-3 decades, but most are now transforming into hybrid solutions, combining the best of cloud (public) and on-premises (private) solutions. The key consideration here is how much of the log data you can collect and transfer to the cloud for Microsoft Sentinel ingestion. We will cover data connectors more in Chapter 3, Managing and Collecting Data.

Active Directory is a key solution that should also be included in this component. It will be extended to public cloud infrastructure (component 09) and addressed in the Privileged Access Management section (component 10). The best defense for Azure Active Directory is to deploy the Microsoft Defender for Identity solution, which Microsoft developed to specifically protect Active Directory domain controllers.

Public Cloud Infrastructure: These solutions are