AI for Encrypted Traffic Analysis

Objective

In this lesson you will learn how to enable and validate Encrypted Traffic Analytics (ETA) using an AIOps platform to analyze encrypted flows without decrypting them. You will configure ETA ingestion, enable AI/ML detection for TLS/SSL fingerprinting, and verify detections. This matters in production because decrypting traffic at scale is often infeasible due to privacy, CPU, or legal constraints — ETA lets you detect malware, command-and-control, and anomalous encrypted flows using flow metadata and ML. Real-world scenario: a global enterprise uses ETA to identify suspicious TLS sessions originating from remote VPN users without terminating TLS at a central site.

Quick Recap

Refer to the topology from Lesson 1 for network layout and device names. This lesson adds the AIOps analytics collector at the management hostname lab.nhprep.com (the AIOps service) and configures it to accept telemetry and flow records from existing firewalls and VPN headends already present in the topology.

Note: No new device IP addressing in this lesson — the AIOps collector is reached at host lab.nhprep.com and managed with the password Lab@123.

Key Concepts

• Encrypted Traffic Analytics (ETA): Uses flow metadata (SNI, JA3/JA3S TLS fingerprints, packet sizes, timing, ciphers) and ML to classify encrypted sessions without decryption. Think of ETA as a microscope that examines the envelope and fingerprints of a sealed package rather than opening it.
• Telemetry and Flow Sources: ETA relies on telemetry sources such as NetFlow/IPFIX, TLS session metadata, and device telemetry. In production, firewalls and VPN headends export these to a central collector for correlation.
• ML-based Detection: The platform compares observed fingerprints and behavioral features against ML models trained to recognize malicious patterns (e.g., beaconing, unusual cipher suites). This reduces false positives compared to simple signature matching.
• Privacy & Compliance: Because ETA does not decrypt payloads, it preserves data privacy and reduces legal exposure; however, SNI and JA3-like fingerprints are still sensitive and should be handled securely.
• Operational Integration: ETA results feed into alerting, policy recommendations (e.g., blocking by application ID), and automation workflows to remediate risky encrypted flows.

Step-by-step configuration

Step 1: Register telemetry sources with the AIOps collector

What we are doing: We register the firewall and VPN headend as telemetry sources so the AIOps collector can receive NetFlow/IPFIX and TLS metadata. This step is essential because without registered sources the collector will not accept or attribute flow telemetry.

# Authenticate and register device: Firewall1
curl -s -k -u admin:Lab@123 -X POST "https://lab.nhprep.com/api/sources" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Firewall1",
    "type": "netflow",
    "hostname": "firewall1.lab.nhprep.com",
    "listen_ip": "0.0.0.0",
    "port": 2055,
    "protocol": "ipfix",
    "collector_group": "eta-collectors"
  }'

# Register device: VPNHeadend1
curl -s -k -u admin:Lab@123 -X POST "https://lab.nhprep.com/api/sources" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "VPNHeadend1",
    "type": "netflow",
    "hostname": "vpnheadend1.lab.nhprep.com",
    "listen_ip": "0.0.0.0",
    "port": 2056,
    "protocol": "ipfix",
    "collector_group": "eta-collectors"
  }'

What just happened: Each curl command authenticates to the AIOps collector at lab.nhprep.com and posts a JSON payload registering a telemetry source. The collector saves source metadata and opens the configured UDP/TCP ports to receive NetFlow/IPFIX telemetry. Protocol-level detail: NetFlow/IPFIX templates determine which fields (such as TLS SNI, flow timestamps, packet sizes) are exported; ensure devices export the required fields.

Real-world note: In production, register each device with a unique certificate or API token instead of plaintext basic auth; also limit the collector's listen scope to specific interfaces.

Verify:

# List registered sources
curl -s -k -u admin:Lab@123 "https://lab.nhprep.com/api/sources"

Expected output:

[
  {
    "id": "src-001",
    "name": "Firewall1",
    "type": "netflow",
    "hostname": "firewall1.lab.nhprep.com",
    "port": 2055,
    "protocol": "ipfix",
    "collector_group": "eta-collectors",
    "status": "active"
  },
  {
    "id": "src-002",
    "name": "VPNHeadend1",
    "type": "netflow",
    "hostname": "vpnheadend1.lab.nhprep.com",
    "port": 2056,
    "protocol": "ipfix",
    "collector_group": "eta-collectors",
    "status": "active"
  }
]

---

Step 2: Enable ETA analytics engine & select ML models

What we are doing: We enable the ETA analytics module and choose the ML model profile appropriate to the environment (e.g., "enterprise-protect" balanced detection). This activates feature pipelines such as JA3/JA3S fingerprint extraction and anomalous flow detection.

# Enable ETA analytics module and set model profile
curl -s -k -u admin:Lab@123 -X POST "https://lab.nhprep.com/api/analytics/eta/enable" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "model_profile": "enterprise-protect",
    "collect_ja3": true,
    "collect_sni": true,
    "sensitivity": "balanced"
  }'

What just happened: The API call activates the ETA pipeline on the collector. Setting collect_ja3 and collect_sni instructs the system to extract TLS fingerprint features from incoming telemetry. The model_profile determines thresholds and feature weights; "balanced" sensitivity aims to minimize false positives while still catching covert malware.

Real-world note: For high-risk environments (e.g., SOC monitoring critical assets), you might choose a higher sensitivity profile, accepting more alerts for greater coverage.

Verify:

# Check ETA status
curl -s -k -u admin:Lab@123 "https://lab.nhprep.com/api/analytics/eta/status"

Expected output:

{
  "eta_enabled": true,
  "model_profile": "enterprise-protect",
  "collect_ja3": true,
  "collect_sni": true,
  "sensitivity": "balanced",
  "pipeline_status": "running"
}

---

Step 3: Configure flow retention and privacy controls

What we are doing: We configure how long flow metadata is retained and enable privacy masking for sensitive fields (e.g., mask full SNI or anonymize client IPs). This balances forensic value and compliance.

# Set retention and privacy settings
curl -s -k -u admin:Lab@123 -X PATCH "https://lab.nhprep.com/api/settings/retention" \
  -H "Content-Type: application/json" \
  -d '{
    "flow_retention_days": 90,
    "mask_sni": false,
    "anonymize_client_ip": true
  }'

What just happened: The collector will retain flow metadata for 90 days. Masking SNI is set to false (full SNI retained); client IPs will be anonymized in stored records. Protocol-level implication: anonymization should preserve prefixes for aggregation while removing exact addresses for privacy.

Real-world note: Legal teams often require shorter retention or full masking in regulated environments; adjust settings accordingly.

Verify:

# Show retention and privacy config
curl -s -k -u admin:Lab@123 "https://lab.nhprep.com/api/settings/retention"

Expected output:

{
  "flow_retention_days": 90,
  "mask_sni": false,
  "anonymize_client_ip": true
}

---

Step 4: Validate incoming telemetry and extracted features

What we are doing: We verify that incoming NetFlow/IPFIX from devices is received and that the ETA pipeline extracts features like JA3 fingerprints and SNI, producing events that can be classified.

# Show latest flows ingested (most recent 5)
curl -s -k -u admin:Lab@123 "https://lab.nhprep.com/api/flows?limit=5&sort=timestamp:desc"

What just happened: The API returns recent flow records ingested by the collector. Each record contains metadata fields such as timestamp, src/dst (anonymized depending on config), SNI, JA3, client_port, server_port, bytes, and duration. At the protocol level, the flow record may have been converted from NetFlow template fields to a normalized internal schema.

Real-world note: If you see no flows, confirm NetFlow/IPFIX exporters are pointed to the collector and that device templates include TLS metadata fields.

Verify:

# Example flow record output (first of 5)
{
  "flow_id": "flow-20260401-0001",
  "timestamp": "2026-04-01T12:34:56Z",
  "src_ip": "10.200.0.0/24", 
  "dst_ip": "93.184.216.34",
  "src_port": 52344,
  "dst_port": 443,
  "protocol": "TCP",
  "bytes": 12500,
  "duration_ms": 2300,
  "sni": "example.lab.nhprep.com",
  "ja3": "771,4865-4866-4867-49195-49199,23-65281-52445,0-1-2,0",
  "classification": "unknown",
  "eta_score": 0.12
}

Expected: You should see flows with populated sni and ja3 fields. The classification may be "unknown", "suspicious", or "malicious" depending on ML verdicts.

---

Step 5: Generate an alert and examine AI insights

What we are doing: Trigger a test flow or simulate an idle VPN session anomaly and review the ETA-generated alert and AI assistant recommendations. This shows how ETA integrates into incident workflows.

# Query recent alerts related to ETA
curl -s -k -u admin:Lab@123 "https://lab.nhprep.com/api/alerts?source=eta&limit=10"

What just happened: The collector returns alerts produced by the ETA engine. Each alert includes a description, affected device(s), probable cause, confidence level, and recommended remediation (e.g., block JA3 hash, apply timeout). The AI assistant may suggest policy changes or next steps.

Real-world note: Automation can be attached to high-confidence alerts to perform remediation (e.g., update firewall rule), but always validate in staging before automated enforcement.

Verify:

# Example alert output (one record)
{
  "alert_id": "alert-20260401-1001",
  "timestamp": "2026-04-01T12:40:10Z",
  "source": "eta",
  "title": "Suspicious TLS JA3 Hash Detected",
  "description": "High-confidence match between observed JA3 and known C2 fingerprint. Affected device: VPNHeadend1",
  "affected_devices": ["VPNHeadend1"],
  "probable_cause": "Beaconing to known malicious JA3 fingerprint",
  "confidence": 0.92,
  "recommended_action": "Quarantine source network; block JA3 hash at edge firewall",
  "status": "new"
}

Expected: Alert contains a clear description, confidence score, affected device list, and recommended remediation workflow.

Verification Checklist

• Check 1: AIOps collector shows registered sources. Verify with GET /api/sources and see Firewall1 and VPNHeadend1 with status "active".
• Check 2: ETA analytics pipeline is running. Verify with GET /api/analytics/eta/status and see "pipeline_status": "running".
• Check 3: Flows contain TLS features (sni, ja3). Verify with GET /api/flows?limit=5 and confirm fields sni and ja3 are present.
• Check 4: ETA has produced alerts for suspicious flows. Verify with GET /api/alerts?source=eta&limit=10 and inspect confidence and recommended_action.

Common Mistakes

Symptom	Cause	Fix
No flows shown in collector output	Devices not exporting NetFlow/IPFIX or wrong collector IP/port	Verify device export config; ensure IP/port matches lab.nhprep.com collector and firewall rules allow UDP/TCP to collector ports
sni/ja3 fields are empty	Export template lacks TLS metadata fields	Update exporter template on device to include TLS SNI and JA3-like fields; check device firmware supports TLS metadata export
ETA pipeline shows "not running"	Analytics module not enabled or models not loaded	Re-run enable API, check ML model deployment logs, ensure sufficient CPU/GPU resources for model inference
Many false positives from ETA	Sensitivity profile too aggressive or model not tuned for environment	Lower sensitivity to "balanced" or "conservative"; whitelist known JA3 hashes for internal apps after investigation

Key Takeaways

• ETA lets you detect malicious encrypted traffic without decrypting payloads by using TLS metadata (SNI, JA3) and ML-based behavioral analysis.
• Proper telemetry export (correct NetFlow/IPFIX templates) and secure registration of flow sources to the collector are prerequisites for ETA to work.
• Configure retention and privacy settings to meet compliance while preserving forensic utility; anonymization and SNI masking are useful controls.
• In production, integrate ETA alerts into automation and incident response workflows cautiously — validate ML recommendations before automated enforcement.

Tip: Treat ETA as a powerful sensor in your security stack. Use it to prioritize investigations and drive policy hygiene (e.g., replace broad port-based rules with app-aware policies) rather than as the sole enforcement mechanism.