BroadSoft Secrets Revealed: Collecting the Garbage

By Scott Hoffpauir, Managing Partner

When we started BroadSoft in 1998, we knew that voice networks were going to transition from circuit switching to packet switching. There were a bunch of companies doing gateways between the two technologies, using them for niche applications. We wanted to look further out, to a time when packet based networks would be mainstream, so we decided to focus on the applications that would go on top of these networks.

As all startups do, we created a really nice deck of PowerPoint slides describing our solution and how it worked. For some reason, the slides are still available on the Internet — here’s a link if you’re interested. Notice the crafty use of animation and the cool purple and teal which we chose as our company colors. With the slides done, we needed to build a product to match our pretty slides.

We hired a bunch of great people and started to lay out our product architecture. We also needed to make decisions on what technology to use. One of the biggest decisions we made, starting out, was selecting Java as our primary programming language. Today, Java is what high schoolers learn in their computer science class, but back then it was nearly brand new. Like any new technology, it had issues and it was a decision that kept me up at night. (The other thing that keeps me up at night is the trifecta of red wine, steak and creme brulee, but that’s a different story) Anyways, here’s my story of our decision to use Java.

Before BroadSoft, we decided to be progressive and innovative at the previous company I was at, so we used Java for a management system providing a GUI to interact with a cellular switching system. It worked great! We were more efficient and had higher quality code. Compared to C and C++, it was so much easier to use and faster to try things out. It always seemed with C++, you either had people that used it like C or people that felt compelled to use every aspect of the language, making it incomprehensible to everyone else.

At BroadSoft, we had two factions on the team — one for Java and one for C/C++, which meant I had to make the decision and I decided to go with Java. Now, at the time, no one was using Java for real-time applications. With a call processing application, you have to have five 9’s availability and millisecond response time, matching the performance characteristics of class 5 switches. Java uses this concept of garbage collection, which means it pauses periodically to clean things up and release memory.

There was concern on the team that garbage collection wouldn’t have the right performance for our application. So, like any great leader, I gave the team a lecture and told them to stop being so negative and get coding. I would later regret that lecture. We started development on our initial product — what today you would call a minimum viable product. Everything was working great, but every now and again, the application would stop for three or four seconds every few hours and do garbage collection.

Now, this doesn’t sound like that big of a deal, but when you’re trying to replace a class 5 switch, you have to match its performance characteristics. Five 9’s reliability means you can only have 326 seconds of downtime a year. A few seconds every few hours was a lot more than that. Holy crap, I panicked. We had just invested three months of development. Starting over using a different programming language wasn’t really an option.

Luckily, we had a great team of developers — Alex Doyle, Maria Doyle, Steve Liao, Dave Block, Wesley Hacker, Sam Hoffpauir, Jamie Palmer. They worked tirelessly to reduce the garbage collection interval. They even worked directly with Sun (where Java came from) to tune the virtual machine. Sun eventually took some of our input and incorporated it into future versions of the virtual machine, making it work better for real-time applications. All in all, it took us about nine months to get things to an acceptable state — about 150–200ms intervals every hour or so. I lost a considerable amount of sleep during this time.

So, what’s the moral of the story? For us, choosing Java enabled us to move faster and do more with the team we had. While it was a risky decision, it ultimately enabled us to stay ahead and surpass our competition. When you’re breaking new ground, you need to think big and project out. While it’s hard to do, you have to think about where things are going to be in five or ten years, and make sure your decisions are as future proof as they can be. Things move fast and it’s better take risks than get left behind.

Next time, I’ll talk about the other big decision which kept me up at night — our selection of SIP as our core voice communications protocol.