Orbit
Xstreami
The Initial Symptom: Monitoring Graphs Randomly Breaking
The client environment was monitored through a jump server connected over a site-to-site IPsec VPN tunnel. At first glance, everything appeared normal:
- VPN tunnel status showed UP
- Routing was functioning correctly
- SSH access was available
- No significant packet loss was reported
However, monitoring graphs continuously showed interruptions and missing data points. Mafiree was brought in to investigate why this was happening. Because the VPN appeared healthy, the initial assumption was that the problem existed somewhere else in the monitoring path. That assumption turned out to be wrong.
VPN Troubleshooting: Why Tunnel Status Can Be Misleading
One of the most common mistakes during VPN troubleshooting is treating tunnel status as proof of tunnel health. A VPN can remain established while still experiencing serious operational problems beneath the surface.
A VPN can remain established while still experiencing:
| Hidden Issue | Visible Impact | Tunnel Status |
|---|---|---|
| Rekeying failures | Intermittent data gaps | Still UP |
| Fragmentation problems | Large packet drops | Still UP |
| Lifetime mismatches | Periodic reconnects | Still UP |
| Phase 2 negotiation failures | SA reset events | Still UP |
| Firmware-related bugs | Config auto-revert | Still UP |
From an application perspective, even brief interruptions can cause monitoring systems to miss data collection windows and create gaps in graphs. The challenge is that these interruptions are often too short to trigger a full tunnel outage alarm.
MTU Validation Across the VPN Tunnel
Because the issue started after VPN-related changes, Mafiree's engineering team considered MTU as an early suspect.
Testing revealed:
- 1472-byte packets — failed (exceeded VPN payload limit)
- 1400-byte packets — succeeded
This confirmed that VPN encapsulation overhead was reducing the available payload size. Further analysis of the IPsec configuration showed:
- AES-256 encryption
- SHA-256 authentication
- IKEv2 with PFS enabled
The estimated ESP overhead reduced the usable VPN payload to approximately 1365 bytes. While this finding was important, it did not fully explain the intermittent monitoring interruptions — smaller packets and control-plane traffic continued to function normally.
Network Path Analysis Using MTR
To eliminate routing instability as a potential cause, Mafiree engineers performed a 100-packet MTR analysis. Results showed:
| Metric | Result | Assessment |
|---|---|---|
| Packet loss | 0% | Clean |
| Latency stability | Normal — no jitter | Clean |
| RTT consistency | Stable across all hops | Clean |
| Routing changes | None observed | Clean |
At this stage, the network path itself appeared completely healthy. The investigation continued.
SSH Performance Testing and Early Findings
Because monitoring relied on communication through the jump server, SSH responsiveness was tested repeatedly. Ten consecutive SSH sessions completed successfully with connection times consistently below one second. This confirmed:
- Authentication was functioning correctly
- Tunnel responsiveness appeared normal
- No obvious connectivity failures were visible
The tunnel appeared healthy, but the service using the tunnel did not. Spot checks passed — but the issue persisted in production.
Discovering a Hidden Timing Pattern
Rather than relying on occasional SSH tests, Mafiree engineers developed a continuous SSH validation script with verbose logging. This approach exposed something that previous testing had completely missed.
Connection failures were not random.
A timing pattern began to emerge. Failures appeared to occur at recurring intervals, suggesting a possible relationship with tunnel lifetime values, rekey operations, or Security Association renegotiation events. When intermittent failures follow a predictable schedule, timing becomes a critical diagnostic clue.
Firewall Firmware Bug: Correlating the Issue
The next step was reviewing the change history, which we examined more closely as the investigation deepened. A timeline comparison revealed that the issue began immediately after the client upgraded firewall firmware and modified VPN settings to align with the new firmware recommendations. This shifted the investigation away from monitoring systems and toward the VPN itself.
The team verified every parameter:
- Phase 1 and Phase 2 encryption settings
- Preshared keys and authentication methods
- Perfect Forward Secrecy configuration
- Lifetime values on both ends
Everything matched. Yet the issue persisted.
Why Matching VPN Parameters Was Not Enough
At this stage, the tunnel configuration appeared correct on both sides. Even after adjusting lifetime values to match the client's settings, intermittent interruptions continued. This was a critical turning point.
Many troubleshooting efforts stop once configuration values appear identical. The investigation continued here because the observed behavior still did not match the expected outcome — and evidence, not assumptions, drove every next step.
Firewall Logs Revealed the Real Problem
The breakthrough came when we reviewed firewall syslog events rather than relying solely on VPN status pages.
Log analysis showed that VPN-related values were being automatically regenerated and reverted to firmware defaults during operation. Although administrators manually corrected the settings, the firewall periodically altered parameters without any visible indication on the dashboard.
This explained everything:
- The tunnel remained established
- Connectivity appeared mostly normal
- Monitoring graphs continued breaking intermittently
The issue was not a traditional tunnel outage. It was a configuration consistency problem occurring beneath the visible VPN status indicators — invisible to standard monitoring tools.
VPN Troubleshooting Resolution
As a temporary mitigation, Mafiree recommended aligning tunnel settings with the values the firewall repeatedly enforced. This improved stability but did not completely eliminate concerns regarding the underlying platform behavior.
The client later replaced the firewall. Following replacement:
- Monitoring stabilized immediately
- VPN interruptions disappeared entirely
- Graph collection returned to normal
- No additional tunnel anomalies were observed
The evidence strongly indicated that the issue was related to firmware behavior and automatic VPN parameter handling — not routing, monitoring infrastructure, or network performance.
See past the "healthy" status indicator.
Get the visibility that catches silent failures before your customers do.
Lessons Learned from the Investigation
-
1. A tunnel being UP does not mean it is healthy
Status indicators only show establishment state. They do not guarantee application stability or configuration consistency.
-
2. Always correlate problems with recent changes
The firmware upgrade timeline became one of the most important clues in the entire investigation. Change history is always worth reviewing first.
-
3. Continuous testing reveals what spot checks miss
Single SSH tests passed repeatedly. Only continuous scripted testing exposed the hidden recurring failure pattern.
-
4. Logs are more valuable than dashboards
VPN dashboards showed a healthy tunnel. Syslogs revealed the firewall was silently reverting its own configuration.
-
5. Evidence-based troubleshooting matters
Every hypothesis was tested and either validated or eliminated through evidence — never through assumption. That discipline is what found the real root cause.
