Building Your AIOps Practice
Objective
In this lesson you will learn how to start an AIOps practice by validating telemetry data collection on a wireless controller, interpreting telemetry subscription state, and using those telemetry signals to deliver quick operational wins. This matters in production because reliable, timely telemetry is the foundation for automated analysis, anomaly detection, and root-cause workflows — without confirmed telemetry feeds an AIOps pipeline has no trustworthy input. Real-world scenario: when you onboard a new telemetry collector into an enterprise AIOps platform, you must verify device connections and subscription health before trusting automated reasoning to escalate client-impacting issues.
Topology
This lesson uses the same topology presented earlier in the lab series. No new routers or switches are added. We focus on the C9800 controller telemetry connection to the telemetry collector.
ASCII topology (only IPs shown where available):
+---------------------------+ +---------------------------+
| POD4 - C9800 - CL1 | | Telemetry Collector |
| (wireless controller) | | (AIOps / collector node) |
| | | |
| source IP: 192.168.4.7 |-----------------------| peer IP: 172.100.1.53 |
| | TCP port: 25103 | |
+---------------------------+ +---------------------------+
Device Table
| Device Name | Role | Relevant IPs / Ports |
|---|---|---|
| POD4 - C9800 - CL1 | Wireless controller (telemetry source) | source address 192.168.4.7 |
| Telemetry Collector | AIOps collector / ingestion endpoint | peer address 172.100.1.53 :25103 |
Tip: This lesson focuses on telemetry verification and interpretation. The collector IP and controller source IP are taken directly from the controller outputs you will examine.
Quick Recap
From Lesson 1 you already saw the environment where the wireless controller (C9800) runs and must export telemetry to a central collector. We are not adding devices in this lesson. Instead, we validate the collector link and the subscription state that feeds AIOps algorithms. The two controller CLI commands used for verification in this lesson are:
sh telemetry connection allsh telemetry ietfsubscription summary
These commands tell you whether the device has an active TCP/TLS connection to the collector and whether the device has telemetry subscriptions configured and valid.
Key Concepts (before hands-on)
- Telemetry connections vs subscriptions: A telemetry connection is the transport session (peer IP, port, source interface/address, and state) between the device and collector. A subscription is a configured data stream specifying what telemetry to send. Both must be healthy for AIOps to receive usable data.
- Subscription counts and capacity: Devices list maximum supported subscriptions (example: 128). If you push more than the supported limit, additional subscriptions will fail — causing blind spots in your AIOps signals.
- State interpretation: Typical states include Active (data flowing), Connecting (handshake in progress; may indicate certificate or firewall issues), and N/A for missing configuration. The controller’s legend helps map textual states to root causes.
- Why transport details matter: The peer IP, port, and source address determine routing, firewall, and VRF considerations. If the telemetry TCP session fails, higher-level AIOps pipelines receive no telemetry even if subscriptions appear configured.
- Real-world implication: In production, a single subscription gap can mean the difference between early detection of a roaming problem and a user-visible outage. Always confirm both connection and subscription health.
Steps
Step 1: Validate the telemetry transport connection
What we are doing: Confirm the controller has an active transport session to the collector. This verifies network path, ACLs, and the collector’s listener are reachable from the controller — the first prerequisite for telemetry ingestion.
POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address Port VRF Source Address State State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109 172.100.1.53 25103 0 192.168.4.7 Active Connection up
What just happened: The command lists telemetry transport sessions the controller has established. The output shows a connection to peer 172.100.1.53 on TCP port 25103, sourced from 192.168.4.7 in VRF 0, and the session is Active (Connection up). At the protocol level this indicates the device completed the transport handshake and can send telemetry frames to the collector.
Real-world note: An Active state means the collector accepted the TCP/TLS session — in production this confirms path, NAT, and firewall rules are correct for that flow.
Verify:
POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address Port VRF Source Address State State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109 172.100.1.53 25103 0 192.168.4.7 Active Connection up
Step 2: Inspect the telemetry subscription summary
What we are doing: Check how many telemetry subscriptions exist and whether they are valid. Subscriptions are what define the actual data streams AIOps consumes (e.g., interface stats, client metrics).
POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All 112 112 0
Dynamic 0 0 0
Configured 112 112 0
Permanent 0 0 0
Active–All good Connecting –Cert/FW issue N/A –Telemetry config missing
What just happened: The device reported there are 112 subscriptions configured and all 112 are valid. The maximum supported subscriptions is 128, so you have headroom. The legend indicates what other states would imply (e.g., Connecting often points to certificate or firewall problems).
Real-world note: If subscriptions show valid but you still see client issues in AIOps, verify the transport connection (Step 1) and collector-side parsing — both ends must be healthy.
Verify:
POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All 112 112 0
Dynamic 0 0 0
Configured 112 112 0
Permanent 0 0 0
Active–All good Connecting –Cert/FW issue N/A –Telemetry config missing
Step 3: Force push telemetry configuration to devices (GUI action) and confirm
What we are doing: When telemetry subscriptions drift or you add collectors en masse, the management plane can push telemetry settings to devices. In the referenced environment this is done via the inventory action that forces an update of telemetry settings; this ensures the device has the intended subscriptions and connection parameters.
- GUI action to perform: Inventory > Actions > Telemetry > Update Telemetry Settings
Why this matters: Pushing settings centrally ensures consistency across the fleet, enabling predictable data quality for AIOps analysis. It’s a fast remediation when devices miss configuration updates.
What just happened: After you click Update Telemetry Settings, the management system pushes the telemetry configuration to the controller(s). The controller will reconcile subscriptions and re-establish connections if needed.
Real-world note: Use “force push” sparingly — in production, schedule pushes during maintenance windows to avoid transient telemetry churn that can confuse downstream analytics.
Verify: Re-run the two show commands to confirm the transport and subscription state after the push.
POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address Port VRF Source Address State State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109 172.100.1.53 25103 0 192.168.4.7 Active Connection up
POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All 112 112 0
Dynamic 0 0 0
Configured 112 112 0
Permanent 0 0 0
Active–All good Connecting –Cert/FW issue N/A –Telemetry config missing
Step 4: Use telemetry insights (Client 360 / Intelligent Capture) for a quick win
What we are doing: Use telemetry-equipped features such as client-centric captures and timelines to deliver fast operational value: root-cause of a roaming failure, client OS driver issues, or poor RSSI detection.
- In the referenced environment the platform shows: "360 Views Client / Client 360 Time Travel Starts the Intelligent Capture Flow" and "Intelligent Capture Client and AP usecases Passive and on-demand packet capture, Anomaly capture and OTA capture Real-time client and AP statistics".
Why this matters: Rather than awaiting long-term trend models, these features provide immediate evidence (packet captures, time-traveled telemetry) you can use to resolve high-impact incidents quickly — the classic AIOps 'quick win'.
What just happened: When you trigger Intelligent Capture from the platform, the collector coordinates with the device to capture packets or telemetry around the time of interest. That capture is then correlated with client and AP telemetry for a faster root-cause.
Real-world note: Intelligent Capture is invaluable for RCA of intermittent wireless failures; enabling it selectively (on customers or high-severity incidents) reduces storage and operational overhead.
Verify: Confirm the controller still has active telemetry connection and valid subscriptions; without those the captured metadata may be incomplete.
POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address Port VRF Source Address State State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109 172.100.1.53 25103 0 192.168.4.7 Active Connection up
POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All 112 112 0
Dynamic 0 0 0
Configured 112 112 0
Permanent 0 0 0
Active–All good Connecting –Cert/FW issue N/A –Telemetry config missing
Step 5: Troubleshoot a "Connecting" state — what to check
What we are doing: If subscriptions or connections show Connecting, map that to common causes and validate the network/PKI components. The device legend indicates Connecting – Cert/FW issue.
POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All 112 111 1
Dynamic 0 0 0
Configured 112 111 1
Permanent 0 0 0
Active–All good Connecting –Cert/FW issue N/A –Telemetry config missing
What just happened: This hypothetical output (illustrative of what you would see) highlights one invalid subscription. An invalid subscription paired with a Connecting connection commonly points to certificate trust issues (TLS handshake failing) or firewall blocking the TLS port on the collector path.
Real-world note: In production, coordinate with security and PKI teams to verify collector certificates and with network teams to check firewall rules for the collector TCP port (e.g., 25103).
Verify: Use the connection show to confirm whether the transport itself is established or stuck in handshake.
POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address Port VRF Source Address State State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109 172.100.1.53 25103 0 192.168.4.7 Connecting TLS handshake in progress
(If you see Connecting, take certificate and firewall steps described above.)
Verification Checklist
- Check 1: Telemetry transport is active — run
sh telemetry connection alland confirm the collector peer (172.100.1.53) appears with State "Active" and Source Address 192.168.4.7. - Check 2: Subscriptions are configured and valid — run
sh telemetry ietfsubscription summaryand confirm All Total == Valid (example: 112 / 112) and Maximum supported is adequate (128). - Check 3: Intelligent Capture readiness — after forcing telemetry config, trigger a capture from the platform and verify telemetry connection remains Active during/after the capture.
Common Mistakes
| Symptom | Cause | Fix |
|---|---|---|
| Telemetry connection missing or not listed | Network path or firewall blocking collector port (25103) or wrong collector IP | Verify routing to 172.100.1.53, ensure firewalls allow TCP 25103, and confirm correct collector IP in device config |
| Subscriptions configured but show Invalid | Certificate mismatch or subscription pushed with bad parameters | Re-push telemetry config from Inventory > Actions > Telemetry > Update Telemetry Settings and verify PKI/trust chain |
| Some subscriptions showing as Connecting | TLS handshake failures, collector not accepting TLS, or intermediate inspection breaking TLS | Validate collector certificate, check DPI/inspection devices on path, and confirm TLS compatibility |
| Telemetry present but AIOps lacks client context | Collector parsing or platform correlation issue | Check collector ingestion logs and mapping rules; ensure the platform's mapping of device IDs to inventory (asset records) is accurate |
Key Takeaways
- Confirm both the transport connection (peer/port/source/state) and the subscription health (total vs valid) — both are required for trustworthy AIOps signals.
- Use central push (Inventory > Actions > Telemetry > Update Telemetry Settings) to keep telemetry configuration consistent across devices; verify success with the controller show commands.
- Quick wins for AIOps come from focused features like Intelligent Capture and Client 360: these provide immediate, actionable evidence when telemetry is healthy.
- When you see non-Active states, treat them as a combined network + PKI troubleshooting problem: route/firewall for transport and certificates/TLS for handshake.
Final note: Reliable telemetry is the metering system of AIOps — invest time upfront to validate connections and subscriptions. If your foundation is sound, automated reasoning and remediation become practical and trustworthy in production.