NHPREP — Hands-On CCNA, CCNP, CCIE Labs & Networking Courses

Objective

In this lesson you will integrate Catalyst Center with an external ITSM system and build automated workflows for CMDB synchronization and incident creation. You will learn how Catalyst Center components coordinate (provisioning-service, orchestration-engine, spf-service, network-validation, network-programmer, workflow-worker and maglev-server), how a provisioning request becomes a device configuration change and an ITSM incident, and how to monitor and troubleshoot those workflows in production. In enterprise environments this is used to ensure automated, auditable change control (CMDB updates + incident creation) whenever automation pushes device configuration — critical for compliance and rapid incident response.

Quick Recap

We keep the topology from lesson 1 (Catalyst Center appliance managing campus devices). This lesson adds an external ITSM server for incident creation and a CMDB sync endpoint. No changes to device interface IPs from Lesson 1 are required.

ASCII topology (management plane):

[Management Switch]                                [ITSM Server]
   mgmt-sw1                                            itsm-server
   mgmt-sw1-mgmt: 10.10.10.2/24               itsm-server-mgmt: 10.10.10.20/24
            |                                          |
            | mgmt VLAN 10.10.10.0/24                  |
            |                                          |
  -----------------------                        -----------------------
  | Catalyst Center VM |------------------------|   ITSM Server VM     |
  |   lab.nhprep.com   |  eth0: 10.10.10.10/24  |   itsm.lab.nhprep.com|
  -----------------------                        -----------------------

All management-plane IPs shown above are used by the workflows in this lesson.

Key Concepts (theory + practical)

Service-oriented workflow model
Catalyst Center decomposes automation into services (provisioning-service, task-service, orchestration-engine, spf-service, network-validation, network-programmer). These services sequence a provisioning request into validation, config generation, push, and post-processing. In practice this lets networks implement multi-stage checks (prechecks → generate config → validation → push) and safely roll back failed changes.
CMDB sync vs. incident creation
A CMDB sync updates a central inventory of devices/config state after automation runs. Incident creation notifies ITSM when an automation event requires human attention or logging. In production, both together provide traceability: a config change in Catalyst Center creates a CMDB entry and the ITSM incident ID ensures auditors can map change to approval/work order.
Workflow orchestration and message flow
When a provisioning request is created: the provisioning-service enqueues a task (task-service); orchestration-engine kicks off the workflow steps and calls spf-service-manager to synthesize device config; network-validation performs pre-checks; network-programmer pushes the config. Each service communicates via the internal message bus; logs from workflow-worker and maglev-server are primary troubleshooting sources.
Observability: logs and package status
The commands magctl appstack status, maglev package status and magctlservice logs are used to verify service health and to trace workflow progress. In production, access to these logs is essential when an ITSM ticket references a failed automated change.
Pre-checks and safety nets
SWIM-style pre-checks (flash, file transfer, entitlement, compatibility) and network validation are necessary before image or config pushes. Always ensure pre-checks are green in a staged environment before production.

Step-by-step configuration

Step 1: Verify Catalyst Center core services are healthy

What we are doing: Confirm the Catalyst Center application stack and service processes are running. This matters because workflows depend on these services to accept provisioning requests, orchestrate tasks, and communicate with external ITSM/CMDB.

magctl appstack status
maglev package status
magctlservice status provisioning-service
magctlservice status orchestration-engine
magctlservice status workflow-worker
magctlservice status maglev-server

What just happened:

magctl appstack status and maglev package status list the overall application stack and installed Maglev packages.
magctlservice status <service> shows whether each microservice is running, enabling the orchestration engine and workflow-worker to process tasks.

Real-world note: If any core service is down, orchestration will not start and REST endpoints will return errors; scheduling maintenance windows solves this for upgrades.

Verify:

magctl appstack status
AppStack: catalyst-center
Status:     RUNNING
Components:
  - maglev-server : RUNNING
  - workflow-worker : RUNNING
  - orchestration-engine : RUNNING
  - provisioning-service : RUNNING
  - task-service : RUNNING
  - spf-service-manager : RUNNING
  - network-validation : RUNNING
  - network-programmer : RUNNING

maglev package status
Package: maglev-core
Version: 2.3.5
Status: ACTIVE
Package: system-updater
Version: 2.3.5
Status: IDLE

magctlservice status provisioning-service
Name: provisioning-service
State: RUNNING
PID: 2345
Uptime: 3 days, 4:12:09

magctlservice status orchestration-engine
Name: orchestration-engine
State: RUNNING
PID: 2401
Uptime: 3 days, 3:58:22

magctlservice status workflow-worker
Name: workflow-worker
State: RUNNING
PID: 2420
Uptime: 2 days, 22:10:45

magctlservice status maglev-server
Name: maglev-server
State: RUNNING
PID: 2300
Uptime: 7 days, 1:01:05

Step 2: Trigger a CMDB sync workflow (provisioning request)

What we are doing: Create a provisioning request that instructs Catalyst Center to generate device config and synchronize inventory back to the CMDB endpoint. This simulates a CMDB sync after an automated deployment.

curl -k -X POST "https://lab.nhprep.com/api/provisioning/v1/requests" \
  -H "Content-Type: application/json" \
  -d '{
    "requestType": "cmdb-sync",
    "source": "automation",
    "devices": [
      { "hostname": "switch1", "managementIp": "10.10.10.101" }
    ],
    "callback": {
      "url": "https://itsm.lab.nhprep.com/api/cmdb/callback",
      "auth": { "user": "nhprep-sync", "password": "Lab@123" }
    }
  }'

What just happened:
The POST submits a provisioning request to provisioning-service. The orchestration-engine will create a task via task-service, then call spf-service-manager to generate device configuration and then call the configured CMDB callback to push inventory changes. The callback section instructs the orchestration to notify the CMDB endpoint on success or failure.

Real-world note: Use secure credentials and ensure the CMDB callback endpoint (itsm.lab.nhprep.com) is reachable from Catalyst Center; firewall rules are a frequent source of failure.

Verify:

# Fetch task list to find the running task
curl -k -X GET "https://lab.nhprep.com/api/task/v1/tasks?filter=requestType:cmdb-sync"

# Expected JSON response
{
  "tasks": [
    {
      "taskId": "task-0001",
      "requestType": "cmdb-sync",
      "status": "IN_PROGRESS",
      "createdBy": "automation",
      "startTime": "2026-04-01T10:12:34Z"
    }
  ]
}

Also verify orchestration logs for CMDB callback processing:

magctlservice logs -r workflow-worker
[2026-04-01T10:12:35Z] INFO workflow-worker: Received task task-0001 for cmdb-sync
[2026-04-01T10:12:36Z] INFO orchestration-engine: Calling spf-service-manager to generate config for switch1
[2026-04-01T10:12:40Z] INFO network-validation: Prechecks passed for switch1
[2026-04-01T10:12:45Z] INFO network-programmer: Successfully pushed configuration to 10.10.10.101
[2026-04-01T10:12:47Z] INFO orchestration-engine: POST https://itsm.lab.nhprep.com/api/cmdb/callback returned 200 (OK)
[2026-04-01T10:12:47Z] INFO workflow-worker: Task task-0001 completed SUCCESS

Step 3: Generate an ITSM incident from a workflow

What we are doing: Demonstrate how the orchestration-engine can create an ITSM incident (for example, to record an automated change or to raise an exception when validation fails). We POST a payload that instructs Catalyst Center to call the ITSM incident endpoint.

curl -k -X POST "https://lab.nhprep.com/api/provisioning/v1/requests" \
  -H "Content-Type: application/json" \
  -d '{
    "requestType": "change-deploy",
    "source": "automation",
    "devices": [
      { "hostname": "switch1", "managementIp": "10.10.10.101" }
    ],
    "itsm": {
      "createIncident": true,
      "endpoint": "https://itsm.lab.nhprep.com/api/incidents",
      "auth": { "user": "nhprep-itsm", "password": "Lab@123" },
      "incidentTemplate": {
        "summary": "Automated change to switch1",
        "priority": "Medium",
        "assignedGroup": "Network Operations"
      }
    }
  }'

What just happened:
This request instructs orchestration-engine to create an ITSM incident using the provided endpoint and credentials. The workflow will call the ITSM API after attempting the deployment (or upon a validation error, depending on orchestration policy). Including the incident template ensures consistent ticket content.

Real-world note: Many organizations require automatic incident creation for unauthorized changes or failed automations; ensure the incident payload meets your ITSM schema.

Verify:

# Check workflow logs for ITSM call and returned incident id
magctlservice logs -r maglev-server
[2026-04-01T11:05:12Z] INFO orchestration-engine: Creating ITSM incident via https://itsm.lab.nhprep.com/api/incidents
[2026-04-01T11:05:13Z] INFO orchestration-engine: ITSM response: {"incidentId":"INC-20260401-5001","status":"CREATED"}
[2026-04-01T11:05:13Z] INFO workflow-worker: Task task-0002 completed SUCCESS (ITSM Incident INC-20260401-5001)

You can also query the ITSM server directly (example):

curl -k -X GET "https://itsm.lab.nhprep.com/api/incidents/INC-20260401-5001" -u nhprep-itsm:Lab@123

Expected ITSM response:

{
  "incidentId": "INC-20260401-5001",
  "status": "OPEN",
  "summary": "Automated change to switch1",
  "priority": "Medium",
  "assignedGroup": "Network Operations",
  "createdAt": "2026-04-01T11:05:13Z"
}

Step 4: Handle and troubleshoot a failed pre-check (validation failure)

What we are doing: Simulate a validation failure and trace the error. This shows how Catalyst Center prevents risky pushes and how the orchestration reports back to ITSM/CMDB.

# Simulate a failing validation by sending a request that triggers network-validation to fail
curl -k -X POST "https://lab.nhprep.com/api/provisioning/v1/requests" \
  -H "Content-Type: application/json" \
  -d '{
    "requestType": "change-deploy",
    "source": "automation",
    "devices": [
      { "hostname": "switch1", "managementIp": "10.10.10.101" }
    ],
    "force": false,
    "validationProfile": "strict"
  }'

What just happened:
This submits a change that the orchestration will validate under a strict profile. If any pre-check fails (SSH unreachable, file transfer problem, config syntax, etc.), network-validation will mark the task as FAILED and orchestration-engine will either abort or create an ITSM incident depending on policy.

Real-world note: A validation failure is often caused by connectivity, credentials, or an unsupported device image; check firewall and credentials first.

Verify:

magctlservice logs -r workflow-worker
[2026-04-01T12:20:11Z] INFO workflow-worker: Received task task-0003 for change-deploy
[2026-04-01T12:20:12Z] INFO network-validation: Starting prechecks for switch1
[2026-04-01T12:20:13Z] ERROR network-validation: SSH unreachable to 10.10.10.101: timeout
[2026-04-01T12:20:13Z] ERROR orchestration-engine: Task task-0003 failed validation, status=FAILED
[2026-04-01T12:20:14Z] INFO orchestration-engine: Posting failure callback to https://itsm.lab.nhprep.com/api/incidents
[2026-04-01T12:20:14Z] INFO orchestration-engine: ITSM response: {"incidentId":"INC-20260401-5002","status":"CREATED"}

You should also see the task status:

curl -k -X GET "https://lab.nhprep.com/api/task/v1/tasks/task-0003"

Expected task response:

{
  "taskId": "task-0003",
  "status": "FAILED",
  "failureReason": "network-validation: SSH unreachable to 10.10.10.101",
  "timestamp": "2026-04-01T12:20:14Z"
}

Step 5: Monitor and collect logs for root cause analysis

What we are doing: Use magctlservice logs and package status commands to collect runtime diagnostics for failed workflows. These are the primary sources for troubleshooting workflow failures and for attaching evidence to ITSM incidents.

magctlservice logs -r workflow-worker
magctlservice logs -r orchestration-engine
magctlservice logs -r maglev-server
maglev package status
# Tail live logs if necessary
magctlservice logs -rf workflow-worker

What just happened:
You collected full logs from workflow-worker, orchestration-engine and maglev-server. -r dumps stored logs; -rf streams live logs for real-time troubleshooting. maglev package status shows whether packages are correctly installed and active; package problems can cause unexpected workflow behaviors.

Real-world note: For large deployments, collect logs and attach to the ITSM incident for faster triage by platform teams.

Verify:

maglev package status
Package: maglev-core
Version: 2.3.5
Status: ACTIVE
Package: workflow-utils
Version: 1.1.0
Status: ACTIVE
Package: system-updater
Version: 2.3.5
Status: IDLE

magctlservice logs -r orchestration-engine
[2026-04-01T12:18:00Z] INFO orchestration-engine: Starting workflow processor
[2026-04-01T12:20:12Z] INFO orchestration-engine: Task task-0003 scheduled
[2026-04-01T12:20:12Z] ERROR orchestration-engine: network-validation failed for task-0003: SSH unreachable
[2026-04-01T12:20:13Z] INFO orchestration-engine: Notifying ITSM endpoint https://itsm.lab.nhprep.com/api/incidents
[2026-04-01T12:20:14Z] INFO orchestration-engine: ITSM returned incidentId INC-20260401-5002

Verification Checklist

Check 1: Core services running — run magctl appstack status and confirm provisioning-service, orchestration-engine, and workflow-worker show RUNNING.
Check 2: CMDB sync completed — POST a cmdb-sync request and confirm workflow status is SUCCESS via curl GET https://lab.nhprep.com/api/task/v1/tasks?filter=requestType:cmdb-sync and look for status IN_PROGRESS → SUCCESS.
Check 3: ITSM incident created — after a change-deploy request confirm orchestration log shows ITSM incidentId and verify incident by querying https://itsm.lab.nhprep.com/api/incidents/{incidentId}.
Check 4: Failed validation handled — simulate a failure and confirm task status is FAILED and an ITSM incident is created.

Common Mistakes

Symptom	Cause	Fix
Workflow stays in IN_PROGRESS and never completes	Firewall or network blocks access from Catalyst Center to device or ITSM endpoint	Open required ports on firewall; verify management IP connectivity and DNS resolution
CMDB callback returns 403 or 401	Incorrect credentials or callback endpoint authentication mismatch	Verify credentials (user: nhprep-sync, password: Lab@123) and endpoint expected auth method
magctlservice reports service DOWN	Service crashed or package update in progress	Check `maglev package status`, review `magctlservice logs -r maglev-server` and restart service if needed during maintenance window
ITSM incident not showing details	Incident schema mismatch or missing required fields in the incidentTemplate	Confirm the incident payload matches ITSM API schema and include required fields (summary, priority, assignedGroup)

Key Takeaways

Catalyst Center uses several cooperating microservices (provisioning-service, orchestration-engine, spf-service-manager, network-validation, network-programmer, workflow-worker, maglev-server) to perform end-to-end automation workflows; verify these are RUNNING before testing automations.
CMDB sync and ITSM incident creation are implemented as callback/integration steps in orchestration flows — this ensures automated changes are recorded and auditable in corporate systems.
Always validate connectivity (firewall, DNS, NTP) and credentials prior to automation runs; most failures are environmental (connectivity or authentication).
Use magctl appstack status, maglev package status, and magctlservice logs (-r / -rf) as your first tools for debugging — collect logs and attach them to ITSM incidents for cross-team troubleshooting.

Tip: In production, use staged environments and enforce strict validation profiles for the first rollback-capable deployments; then relax policies for low-risk, repeatable tasks to improve automation velocity.

ITSM Integration and Workflows

Objective

Quick Recap

Key Concepts (theory + practical)

Step-by-step configuration

Step 1: Verify Catalyst Center core services are healthy

Step 2: Trigger a CMDB sync workflow (provisioning request)

Step 3: Generate an ITSM incident from a workflow

Step 4: Handle and troubleshoot a failed pre-check (validation failure)

Step 5: Monitor and collect logs for root cause analysis

Verification Checklist

Common Mistakes

Key Takeaways