Internet downtime sucks. Period. 

Unfortunately, outages are a fact of modern life. It’s not really a matter of “if” a disruption happens, it’s more like “when.” 

The question is, what does your ISP do when it happens?

Some ISPs use off-the-shelf monitoring systems that only check for hardware or software issues once every 15 or 20 minutes, and many don’t even monitor their active fiber or cables. With that much potential lag time, customers experience the problem long before their ISP’s monitoring system picks it up, meaning that customers end up blowing up the ISP’s notifications before they even know something is wrong.

You can’t blame customers for rage-tweeting when they can’t connect. When your business internet goes out, every minute feels like a lifetime. Ain’t nobody got time for that!  Frustration aside, internet outages can cost a small business more than $50,000 each year. Yikes!

At Pilot, we prioritize keeping our customers connected at all costs, so we continuously upgrade and iterate on our network to prevent all outages that are within our control.

For everything outside of our control, we also invest heavily in tools and systems to make sure we can respond ASAP to get you back online. And that starts with identifying problems faster than you can click “Compose tweet.”

This starts with our Real-Time Fiber Monitoring, which automatically identifies any outages or issues throughout the underground fiber system. (Not to brag, but it’s kind of a big deal.) We also created a proprietary ONT monitoring software, which gives us the same level of insight into potential issues with any of our customers’ onsite hardware.

We’re alerted to points of failure within 15 seconds—not minutes. We then use OSPinsight to pinpoint the exact location of any fiber issues, see the effect on the entire system, and plan any rerouting, if needed. 

Our systems also apply real-time logic to tell us what kind of issue it may be and whether it's impacting one customer, multiple customers, partial building, full building, or a set of buildings. We’ve built that logic so we know what we're dealing with, and can act accordingly, faster. 

Without these integrated systems, that same process would take hours. Literally. In the case of a fiber break, technicians would have to wait until someone can give them access to the data center and the customer location, and then use a handheld device to generate a file, and then send that file back to their engineer, who then has to plot it on mapping software to triangulate the location of the break.

Thanks to our investment in real-time fiber monitoring, our smart use of industry-leading tools, and our proprietary ONT monitoring software, we can skip all that. We get real-time notifications and access to all the relevant information, which allows us to arrive at the scene up to 50% faster, dramatically shortening repair times for our customers.

Real-Time Fiber Monitoring, IRL

On the morning of December 8, 2019,  we received an alert that 11 buildings with ~70 of our business internet customers had lost service. We identified a precise common point of failure (Thanks, real-time fiber monitoring!) and dispatched a technician to the scene almost immediately.

When our technician arrived, he discovered that a fire inside a manhole had destroyed numerous cables, including those of several other internet service providers.

This was a particularly bad underground fire, with more damage than normal, but these types of underground fires happen way more often than you probably think... There are more than 246,000 manholes and service boxes in NYC, and FDNY responds to an average of 3 manhole cover explosions per day every January and February. 

(No, that’s not a typo: THREE PER DAY. Apparently, cockroaches, rats, and rusted-out sidewalk cellar doors didn’t provide enough street-level danger, so we also have to worry about manholes exploding at random. Cool.)

But since Pilot has invested in tools and systems that allow us to respond to unpredictable disruptions—like manhole explosions and underground fires—we’re ready to take immediate action to get our customers back online.

And that response time is key when you’re dealing with an outage that affects multiple ISPs because the first internet service provider on-site is generally the first allowed to go underground to make repairs. 

On that specific Sunday, though, the high-voltage circuit was still active, meaning nobody was allowed in. At least not until the electric company’s technicians could assess the damage, address the downed cables, and ensure it was safe to enter. 

We had two choices: We could wait an undetermined amount of time for ConEd to give us the go-ahead, or we could try to find another way to get our customers back online…

Going the Distance

In case it’s not obvious yet, we really don’t like to keep you waiting.

Using one of our tools, OSPInsight, we examined our network in its entirety and analyzed detailed information on how all of our existing fibers were being used. With real-time access to data about each end-user, location address, fiber priority, endpoint, and their current status, it was easy to see who would be impacted by any rerouting decisions.

We designed a rerouting scenario that avoided the mangled mess inside the affected manhole, and our technicians got to work laying 3,000 feet of new cable.

Sure, doing this required more work than waiting around would have. But having the technology to see this solution and the willingness to go the extra mile* helped us restore service to all of our customers days before the other providers were able to.
(*3,000 feet is actually 0.56818 of a mile, but you get the idea.)

We know every minute counts when your business is experiencing an outage, which is why we continuously invest in ways to prevent customer downtime and accelerate technician response time. Find out if Pilot is in your building and join the thousands of businesses—including Casper, Shake Shack, and Business Insider—who have peace of mind knowing Pilot is helping to keep their business internet up and running.