Manufacturing & Technology
Untangling the Spaghetti Monster
A ground-up infrastructure overhaul for a leader in industrial 3D printing — embedded for months, rebuilt over a single holiday
Before
AfterAt a Glance
- Client
- MakerBot — a leader in industrial 3D printing and additive manufacturing
- Problem
- A production environment running 40 virtual machines across a dozen departments — on undocumented infrastructure that nobody fully understood — that had grown too complex and too critical to fix without first spending months learning exactly how it worked.
- Services
- Infrastructure assessment and full documentation · embedded on-site consulting · VMware and storage architecture · server procurement and build · network remediation · backup strategy with offsite replication · cable management · vendor coordination (VMware, Dell, Cisco, APC)
- Platforms
- VMware ESXi · Cisco Catalyst switching · APC Symmetra UPS · Dell server hardware · shared SAN storage
- Outcome
- 400% throughput increase. Complete system redundancy. Offsite backup replication. New servers built from the ground up. A fully documented environment — and a server room no one is ashamed of anymore.
The Discovery
A spaghetti monster in the server room
The COO was at a holiday party when the network went down. An IT staff member reached him and suggested he come take a look at the server room. He had never been in it before.
What he found was a wall of tangled cables, unlabeled patch panels, and equipment arranged in no particular order. A spaghetti monster. He had always trusted that the infrastructure was broadly under control. Standing in that room, he understood that it wasn't — and that it hadn't been for some time.
The network issues that had been blamed on the ISP for years were not ISP issues. VLAN assignments were undocumented. Ports in the patch panels weren't connected to anything. 10G interfaces capable of driving the throughput the business needed sat running at a fraction of their capacity. There was no system redundancy. Any hardware failure would take down services across the company with no clear path to recovery.
He called us.
“Ports in the patch panels weren't connected to the network. Troubleshooting required extensive guesswork about network configurations.”
— Senior IT Support Engineer
The Scope
Forty virtual machines and no map
The scale of the environment was significant — and the absence of documentation made it more daunting than it needed to be.
The infrastructure ran multiple VMware ESXi hosts with shared SAN storage, supporting 40 virtual machines that spanned every function in the business: mechanical engineering, electrical engineering, software development, materials science, marketing, sales, customer support, finance, and legal. The network ran on Cisco Catalyst switches. Power was backed by APC Symmetra UPS units — but with no redundancy designed into the system, a UPS failure was a production failure.
Every department depended on this environment. None of them knew how it actually worked. Neither, in a meaningful sense, did anyone else — because it had never been properly documented.
Before we could fix it, we had to learn it.
The Approach
Embedded for months before touching a thing
We didn't begin by making changes. We began by becoming the people who understood the environment better than anyone else.
For months, we worked embedded within their team — on-site, inside the business, alongside their IT staff. We traced every cable. We documented every virtual machine, every VLAN, every storage volume, every dependency. We talked to the vendors: VMware, Dell, Cisco, APC — working through configurations and capabilities with the people who built the systems, making sure our understanding was correct before we committed it to paper.
By the time we were done with the assessment, we were the authoritative experts on their infrastructure. We knew it better than it had ever been known. And with that foundation, we could plan a complete overhaul — not one that required guesswork, but one that was designed down to the individual cable run.
The Execution
Taking Humpty Dumpty apart over Christmas
The execution window was the Christmas holiday closure. With the entire company away, we had a clear runway to do the work without disruption to production operations.
We took the environment completely apart and rebuilt it from scratch.
New servers, procured and configured to spec. The virtual machine estate rebuilt on a properly architected VMware foundation with redundancy designed in from the start. Storage arrays reconfigured. A new backup strategy implemented — with offsite replication to a nearby datacenter, so that a failure on-site was no longer also a data loss event. Network remediation throughout: VLAN assignments rationalized, documented, and consistent. Cisco Catalyst switching properly configured to actually use the 10G capacity the hardware had always been capable of.
And the cables — every run organized, dressed with braided sleeves and velcro, labeled, and traced. Not for aesthetics, but because an environment you can read is an environment you can troubleshoot, and an environment you can troubleshoot is one that never has to be a mystery again.
Outcomes
What we delivered.
- 400% throughput increase — immediate, on the first morning back after the holiday
- Full utilization of 10G switching that had been running at a fraction of its capacity for years
- Complete system redundancy — no single points of failure across compute, storage, or power
- Offsite backup replication to a nearby datacenter — a site failure is no longer a data loss event
- New server hardware, built and configured from the ground up
- 40 virtual machines rebuilt on a properly architected VMware foundation
- Consistent, documented VLAN assignments throughout the environment
- Full infrastructure documentation — for the first time in the company's history
- A cable plant anyone on the team can read and trace
- "Now set up for the future, and no longer ashamed to show people the server room." — COO
- "The immediate thing I noticed was I was a hell of a lot less anxious in there." — Senior IT Support Engineer