Oxide Computer Details Service Processor Debugging
TL;DR. Oxide Computer Company encountered and debugged critical service processor failures within their data center rack system, losing network access and control. - The issue involved a disappearing Service Processor losing network connectivity, hindering debugging. - Investigation revealed software bugs related to task starvation and potential stack overflows in the custom Hubris OS. - Engineers implemented debug changes like LED blinking and adjusted task restart times to isolate the root cause.
- Oxide's Service Processor in their Cosmo sleds dropped off the network, making remote diagnostics impossible.
- Debugging efforts focused on Hubris OS, a custom operating system, specifically looking into task starvation and stack overflows.
- Hardware diagnostics and software modifications, including LED indicators, were crucial for issue reproduction and analysis.
Sources
- A disappearing Service Processor (2025) — oxide.computer