Automating major incident management

Problem

Managing major incidents for P1 and P2 tickets can be a complex, time-sensitive process requiring quick communication, root cause assignment and consistent updates across multiple teams. Manual handling of these tasks often leads to delays, miscommunication and increased operational strain, which negatively impacts response time and resolution efficiency. Without automation, coordinating team efforts and ensuring timely updates in IT service management (ITSM) platforms like ServiceNow becomes cumbersome and prone to errors, leading to extended downtime and reduced productivity.

Solution

Nexthink provides a step-by-step approach to automating major incident management (MIM) for P1 and P2 tickets by integrating with an ITSM solution such as ServiceNow. By automating key tasks—such as notifying MIM team members, assigning root causes to the resolving team and updating ServiceNow with relevant details—this integration streamlines operations, reduces manual effort and enhances team coordination. While this guide focuses on ServiceNow, it serves as an example and can inspire similar integrations with other IT service management (ITSM) tools.

Prerequisites

Before you start integrating ServiceNow with Workflows, ensure you have the following:

Nexthink license
ServiceNow integrated with Nexthink
Administrator permissions

Integration flow

This flow covers automating the first part of the MIM notification process to ensure immediate awareness of P1 or P2 tickets and collaboration to find the root cause.

Step 1: Create a ticket

To create a priority P1 or P2 ticket:

In the SNOW Flow Designer, go to the Flow Designer.
Click on Create New and, select Flow.

In the Flow properties, fill in the relevant properties and click Submit.

Define the trigger for the flow in the Flow Designer.

Step 2: Retrieve MIM group members and devices

Once the P1 or P2 ticket is created, ServiceNow retrieves MIM group members from Active Directory (AD) via Lightweight Directory Access Protocol (LDAP) synchronization, checks the Configuration Management Database (CMDB) for assigned devices, and gathers the necessary resources.

To retrieve MIM group members:

In the SNOW Flow Designer, go to the Flow Designer.
Click on Create New and, select Action.

In the Action properties, fill in the relevant properties and click Submit.

Click the + (plus) icon to create a script to retrieve the users from the group, and their devices.

Script example:

(function execute(inputs, outputs) {
   var deviceList = [];
   var j=0;
   for (i=0; i<8; i++){
		deviceList[i] = "empty"
	}
	// Get the users from the group
	var gr = new GlideRecord('sys_user_grmember');
	gr.addEncodedQuery('group=4dcd9658870c465029c687f60cbb356e');
	gr.query();
	while(gr.next()) {
		// Get the computers from the users
		var gr2 = new GlideRecord('cmdb_ci_computer');
		gr2.addQuery('assigned_to',gr.user);
		gr2.addQuery("u_testing_usage","false");
		gr2.addActiveQuery();
		gr2.query();
		while(gr2.next())
		{
			deviceList[j]= gr2.name;
			j++;
		}        
	}			
	// Get the specified count and time frame in seconds
	var specifiedCount = 2;
	var timeFrameSeconds = 120;

	// Get the current date and time in UTC format
	var now = new GlideDateTime();

	// Create a new GlideDateTime for the time frame limit (120 seconds ago)
	var timeFrameLimit = new GlideDateTime();
	timeFrameLimit.addSeconds(-timeFrameSeconds);

	// Query for P1 and P2 incidents created within the time frame
	var incidentGr = new GlideRecord('incident');
	var q1 = incidentGr.addQuery('sys_created_on', '>=', timeFrameLimit);
	var q2 = incidentGr.addQuery('priority',1);
	q2.addOrCondition('priority',2);
	incidentGr.query();

	// Save the devices as the first output
	outputs.devices = deviceList;
	
	// Get the incident Sys ID
	var sysid = incidentGr.sys_id;
	while (incidentGr.next()){
		sysid = incidentGr.sys_id;
	}
	// Save the Sys ID as the second output
	outputs.sysid = sysid
})(inputs, outputs);

The script above saves the list of user devices in an array. This array is set as the output in the script’s Output Variables section. Additionally, the Sys ID of the incident is saved as another output.

Step 3: Generate an API Authentication token

To obtain the token required for API transactions with Nexthink, create a REST step by filling in the following fields:

Connection: Define Connection Inline.
Base URL: https://<instancename>.api.<location>.nexthink.cloud
Resource Path: /api/v1/token
HTTP Method: POST
Headers:
- Authorization: Basic <Base64 encoded clientId:clientSecret>
- To get this value, encode your clientId:clientSecret in Base64 format.
Click on the top right icon to save the fields.

Step 4: Send device information to Nexthink API

ServiceNow retrieves the device names from the CMDB and sends data to the Nexthink API, which collects the Collector IDs associated with those devices from Nexthink. It then sends this information back to ServiceNow to update the ticket.

Create a second REST step with the following fields:

Connection: Define Connection Inline.
Base URL:https://<instancename>.api.<location>.nexthink.cloud
Resource Path:/api/v2/nql/execute
HTTP Method: POST
Headers:
- Accept: application/json,text/csv
- Authorization:
  - Use the Data Picker—click on the icon next to the field—and select 2 – Rest Step > Response Body.
  - Apply the following filters by clicking the Fx icon:
    String > Split
    Separator: "access_token":
    Utilities > Get Last Item from Array
    String > Split
    Separator: ","scope"
    **Utilities > Get First Item from Array
- Content-Type: application/json
Request Content: Add the body content needed for the Nexthink API query execution—e.g., NQL query, or other.

Request body example:

{
"queryId": "<#yourqueryId>",
"parameters":{
"device0": "(here use the data picker to select 1 - Script Step > devices and apply the filter Get Item from Array with the value 0)",
"device1": "(here use the data picker to select 1 - Script Step > devices and apply the filter Get Item from Array with the value 1)",
"device2": "(here use the data picker to select 1 - Script Step > devices and apply the filter Get Item from Array with the value 2)",
"device3": "(here use the data picker to select 1 - Script Step > devices and apply the filter Get Item from Array with the value 3)",
"device4": "(here use the data picker to select 1 - Script Step > devices and apply the filter Get Item from Array with the value 4)",
"device5": "(here use the data picker to select 1 - Script Step > devices and apply the filter Get Item from Array with the value 5)",
"device6": "(here use the data picker to select 1 - Script Step > devices and apply the filter Get Item from Array with the value 6)",
"device7": "(here use the data picker to select 1 - Script Step > devices and apply the filter Get Item from Array with the value 7)",
}

Execute an NQL API Query in Nexthink that retrieves a list of the Collector UIDs for the devices sent. Use the following NQL query code:

devices during past 7d
| where device.name in [$device0,$device1,$device2,$device3,$device4,$device5,$device6,$device7]
| list collector.uid

Step 5: Trigger the Nexthink MIM workflow

Once ServiceNow (SNOW) retrieves the Collector IDs of the devices belonging to MIM team members, the system triggers the Nexthink MIM workflow by sending this data through the workflow API.

Create a new REST step with the following fields:

Connection: Define Connection Inline.
Base URL: https://<instancename>.api.<location>.nexthink.cloud
Resource Path:/api/v2/nql/execute
HTTP Method: POST
Headers:
- Accept: application/json,text/csv
- Authorization:
  - Use the Data Picker—click on the icon next to the field—and select 2 – Rest Step > Response Body.
  - Apply the following filters:
    String > Split
    Separator: "access_token":
    Utilities > Get Last Item from Array
    String > Split
    Separator: ","scope"
    Utilities > Get First Item from Array
- Content-Type: application/json
Request Body: Add the body content needed for the Nexthink API request.

Request body example:

{
"workflowId": "<#yourworkflowId>",
"devices":[
Use the data picker to select 3 - REST Step > Response Body and apply the filter Replace String twice. The first one put “device.collector.uid” and empty as the regex and replace strings, and the second one put \”\n\” and \”, \” (This will translate the output to the collector Ids comma separated).
],
"params":[
Use the data picker to select 1 - Script Step > sysID
]
}

Step 6: Notify MIM team members

Once the system triggers the flow, Nexthink sends an urgent Campaign to MIM team member devices, ensuring they are notified and can take immediate action to resolve the issue.

The Campaign includes the following:

General information: Details about the incident or issue.
Link to join the war room: A direct link for team members to join the virtual war room for real-time collaboration.
Two options for response:
- Acknowledge: Confirms receipt of the message and participation.
  - If the MIM team member clicks on Acknowledge, Nexthink assumes they have used the link to join the War Room call. After, the system sends another urgent campaign where team members can choose from a list of possible root causes of the P1/P2 incident.
  - Once the MIM team member selects an option in the campaign, the workflow triggers a conditional check based on their response. It then calls the ServiceNow (SNOW) API to move the initial P1/P2 ticket to the Resolving Team's queue.
Remind me Later: Allows team members to be reminded later.
- If an MIM team member selects Remind me Later, the workflow waits until the set time ends and then resends the initial campaign.
- The workflow waits until the MIM team member acknowledges the campaign or takes action before the team proceeds with the next steps.
- If the MIM team member selects Remind me Later a second time, the workflow waits until the set time ends and sends a Final Call campaign that includes only one option: Acknowledge.

Last updated 2 months ago