Lesson 6 of 6

Building Your AIOps Practice

Objective

In this lesson you will learn how to start an AIOps practice by validating telemetry data collection on a wireless controller, interpreting telemetry subscription state, and using those telemetry signals to deliver quick operational wins. This matters in production because reliable, timely telemetry is the foundation for automated analysis, anomaly detection, and root-cause workflows — without confirmed telemetry feeds an AIOps pipeline has no trustworthy input. Real-world scenario: when you onboard a new telemetry collector into an enterprise AIOps platform, you must verify device connections and subscription health before trusting automated reasoning to escalate client-impacting issues.

Topology

This lesson uses the same topology presented earlier in the lab series. No new routers or switches are added. We focus on the C9800 controller telemetry connection to the telemetry collector.

ASCII topology (only IPs shown where available):

           +---------------------------+                       +---------------------------+
           | POD4 - C9800 - CL1        |                       | Telemetry Collector       |
           | (wireless controller)     |                       | (AIOps / collector node)  |
           |                           |                       |                           |
           | source IP: 192.168.4.7    |-----------------------| peer IP: 172.100.1.53     |
           |                           |  TCP port: 25103      |                           |
           +---------------------------+                       +---------------------------+

Device Table

Device NameRoleRelevant IPs / Ports
POD4 - C9800 - CL1Wireless controller (telemetry source)source address 192.168.4.7
Telemetry CollectorAIOps collector / ingestion endpointpeer address 172.100.1.53 :25103

Tip: This lesson focuses on telemetry verification and interpretation. The collector IP and controller source IP are taken directly from the controller outputs you will examine.

Quick Recap

From Lesson 1 you already saw the environment where the wireless controller (C9800) runs and must export telemetry to a central collector. We are not adding devices in this lesson. Instead, we validate the collector link and the subscription state that feeds AIOps algorithms. The two controller CLI commands used for verification in this lesson are:

  • sh telemetry connection all
  • sh telemetry ietfsubscription summary

These commands tell you whether the device has an active TCP/TLS connection to the collector and whether the device has telemetry subscriptions configured and valid.

Key Concepts (before hands-on)

  • Telemetry connections vs subscriptions: A telemetry connection is the transport session (peer IP, port, source interface/address, and state) between the device and collector. A subscription is a configured data stream specifying what telemetry to send. Both must be healthy for AIOps to receive usable data.
  • Subscription counts and capacity: Devices list maximum supported subscriptions (example: 128). If you push more than the supported limit, additional subscriptions will fail — causing blind spots in your AIOps signals.
  • State interpretation: Typical states include Active (data flowing), Connecting (handshake in progress; may indicate certificate or firewall issues), and N/A for missing configuration. The controller’s legend helps map textual states to root causes.
  • Why transport details matter: The peer IP, port, and source address determine routing, firewall, and VRF considerations. If the telemetry TCP session fails, higher-level AIOps pipelines receive no telemetry even if subscriptions appear configured.
  • Real-world implication: In production, a single subscription gap can mean the difference between early detection of a roaming problem and a user-visible outage. Always confirm both connection and subscription health.

Steps

Step 1: Validate the telemetry transport connection

What we are doing: Confirm the controller has an active transport session to the collector. This verifies network path, ACLs, and the collector’s listener are reachable from the controller — the first prerequisite for telemetry ingestion.

POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address           Port  VRF  Source Address      State         State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109   172.100.1.53             25103 0     192.168.4.7             Active        Connection up

What just happened: The command lists telemetry transport sessions the controller has established. The output shows a connection to peer 172.100.1.53 on TCP port 25103, sourced from 192.168.4.7 in VRF 0, and the session is Active (Connection up). At the protocol level this indicates the device completed the transport handshake and can send telemetry frames to the collector.

Real-world note: An Active state means the collector accepted the TCP/TLS session — in production this confirms path, NAT, and firewall rules are correct for that flow.

Verify:

POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address           Port  VRF  Source Address      State         State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109   172.100.1.53             25103 0     192.168.4.7             Active        Connection up

Step 2: Inspect the telemetry subscription summary

What we are doing: Check how many telemetry subscriptions exist and whether they are valid. Subscriptions are what define the actual data streams AIOps consumes (e.g., interface stats, client metrics).

POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All      112     112   0
Dynamic  0       0     0
Configured 112   112   0
Permanent 0      0     0
Active–All good    Connecting –Cert/FW issue    N/A –Telemetry config missing

What just happened: The device reported there are 112 subscriptions configured and all 112 are valid. The maximum supported subscriptions is 128, so you have headroom. The legend indicates what other states would imply (e.g., Connecting often points to certificate or firewall problems).

Real-world note: If subscriptions show valid but you still see client issues in AIOps, verify the transport connection (Step 1) and collector-side parsing — both ends must be healthy.

Verify:

POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All      112     112   0
Dynamic  0       0     0
Configured 112   112   0
Permanent 0      0     0
Active–All good    Connecting –Cert/FW issue    N/A –Telemetry config missing

Step 3: Force push telemetry configuration to devices (GUI action) and confirm

What we are doing: When telemetry subscriptions drift or you add collectors en masse, the management plane can push telemetry settings to devices. In the referenced environment this is done via the inventory action that forces an update of telemetry settings; this ensures the device has the intended subscriptions and connection parameters.

  • GUI action to perform: Inventory > Actions > Telemetry > Update Telemetry Settings

Why this matters: Pushing settings centrally ensures consistency across the fleet, enabling predictable data quality for AIOps analysis. It’s a fast remediation when devices miss configuration updates.

What just happened: After you click Update Telemetry Settings, the management system pushes the telemetry configuration to the controller(s). The controller will reconcile subscriptions and re-establish connections if needed.

Real-world note: Use “force push” sparingly — in production, schedule pushes during maintenance windows to avoid transient telemetry churn that can confuse downstream analytics.

Verify: Re-run the two show commands to confirm the transport and subscription state after the push.

POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address           Port  VRF  Source Address      State         State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109   172.100.1.53             25103 0     192.168.4.7             Active        Connection up
POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All      112     112   0
Dynamic  0       0     0
Configured 112   112   0
Permanent 0      0     0
Active–All good    Connecting –Cert/FW issue    N/A –Telemetry config missing

Step 4: Use telemetry insights (Client 360 / Intelligent Capture) for a quick win

What we are doing: Use telemetry-equipped features such as client-centric captures and timelines to deliver fast operational value: root-cause of a roaming failure, client OS driver issues, or poor RSSI detection.

  • In the referenced environment the platform shows: "360 Views Client / Client 360 Time Travel Starts the Intelligent Capture Flow" and "Intelligent Capture Client and AP usecases Passive and on-demand packet capture, Anomaly capture and OTA capture Real-time client and AP statistics".

Why this matters: Rather than awaiting long-term trend models, these features provide immediate evidence (packet captures, time-traveled telemetry) you can use to resolve high-impact incidents quickly — the classic AIOps 'quick win'.

What just happened: When you trigger Intelligent Capture from the platform, the collector coordinates with the device to capture packets or telemetry around the time of interest. That capture is then correlated with client and AP telemetry for a faster root-cause.

Real-world note: Intelligent Capture is invaluable for RCA of intermittent wireless failures; enabling it selectively (on customers or high-severity incidents) reduces storage and operational overhead.

Verify: Confirm the controller still has active telemetry connection and valid subscriptions; without those the captured metadata may be incomplete.

POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address           Port  VRF  Source Address      State         State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109   172.100.1.53             25103 0     192.168.4.7             Active        Connection up
POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All      112     112   0
Dynamic  0       0     0
Configured 112   112   0
Permanent 0      0     0
Active–All good    Connecting –Cert/FW issue    N/A –Telemetry config missing

Step 5: Troubleshoot a "Connecting" state — what to check

What we are doing: If subscriptions or connections show Connecting, map that to common causes and validate the network/PKI components. The device legend indicates Connecting – Cert/FW issue.

POD4 -C9800 -CL1#sh telemetry ietfsubscription summary
Subscription Summary
====================
Maximum supported: 128
Subscription Total Valid Invalid
-----------------------------------------------
All      112     111   1
Dynamic  0       0     0
Configured 112   111   1
Permanent 0      0     0
Active–All good    Connecting –Cert/FW issue    N/A –Telemetry config missing

What just happened: This hypothetical output (illustrative of what you would see) highlights one invalid subscription. An invalid subscription paired with a Connecting connection commonly points to certificate trust issues (TLS handshake failing) or firewall blocking the TLS port on the collector path.

Real-world note: In production, coordinate with security and PKI teams to verify collector certificates and with network teams to check firewall rules for the collector TCP port (e.g., 25103).

Verify: Use the connection show to confirm whether the transport itself is established or stuck in handshake.

POD4 -C9800 -CL1#sh telemetry connection all
Telemetry connections
Index Peer Address           Port  VRF  Source Address      State         State Description
----- -------------------------- ----- ----- ----------------------------- ---------- --------------------
109   172.100.1.53             25103 0     192.168.4.7             Connecting    TLS handshake in progress

(If you see Connecting, take certificate and firewall steps described above.)

Verification Checklist

  • Check 1: Telemetry transport is active — run sh telemetry connection all and confirm the collector peer (172.100.1.53) appears with State "Active" and Source Address 192.168.4.7.
  • Check 2: Subscriptions are configured and valid — run sh telemetry ietfsubscription summary and confirm All Total == Valid (example: 112 / 112) and Maximum supported is adequate (128).
  • Check 3: Intelligent Capture readiness — after forcing telemetry config, trigger a capture from the platform and verify telemetry connection remains Active during/after the capture.

Common Mistakes

SymptomCauseFix
Telemetry connection missing or not listedNetwork path or firewall blocking collector port (25103) or wrong collector IPVerify routing to 172.100.1.53, ensure firewalls allow TCP 25103, and confirm correct collector IP in device config
Subscriptions configured but show InvalidCertificate mismatch or subscription pushed with bad parametersRe-push telemetry config from Inventory > Actions > Telemetry > Update Telemetry Settings and verify PKI/trust chain
Some subscriptions showing as ConnectingTLS handshake failures, collector not accepting TLS, or intermediate inspection breaking TLSValidate collector certificate, check DPI/inspection devices on path, and confirm TLS compatibility
Telemetry present but AIOps lacks client contextCollector parsing or platform correlation issueCheck collector ingestion logs and mapping rules; ensure the platform's mapping of device IDs to inventory (asset records) is accurate

Key Takeaways

  • Confirm both the transport connection (peer/port/source/state) and the subscription health (total vs valid) — both are required for trustworthy AIOps signals.
  • Use central push (Inventory > Actions > Telemetry > Update Telemetry Settings) to keep telemetry configuration consistent across devices; verify success with the controller show commands.
  • Quick wins for AIOps come from focused features like Intelligent Capture and Client 360: these provide immediate, actionable evidence when telemetry is healthy.
  • When you see non-Active states, treat them as a combined network + PKI troubleshooting problem: route/firewall for transport and certificates/TLS for handshake.

Final note: Reliable telemetry is the metering system of AIOps — invest time upfront to validate connections and subscriptions. If your foundation is sound, automated reasoning and remediation become practical and trustworthy in production.