Cloud Therapy: EP 009 – The Time James Cancel Learned How to Prepare for a VoIP Install

Today, James Cancel is in charge of Globalgig’s (formerly Velis4) N.O.C. in Los Angeles but prior to working at Velis4, James was once a Hosted PBX repair engineer… and boy did he learn a painful technical lesson once on a large hosted VoIP installation. So much so that the Chief of Police got involved!

Want to hear more Cloud Therapy? Find us on iTunes and Stitcher!

Browse customer reviews on the best hosted VoIP service providers.

There are thousands of hosted VoIP providers for business. Don’t even try to research it. Just click below and skip to your top 3.

MY TOP 3   

For a full transcript, see below:

    Mike: Do you think at some point in your future you’re going to do another VoIP install, like, hosted VoIP or any type of VoIP? I think you are. I think that’s safe to say. So, this podcast is fantastic. We’ve got a guest on the show, his name is James Cancel. He’s a senior technical specialist for Velis4. Not only is he a technical guy, he’s in charge of Velis4’s entire NOC right now down here in Southern California. He’s got a ton of experience with previous companies as a hosted VoIP engineer and support specialist. Today, he’s going to share with you a hosted VoIP install that went south when he was working as a repair engineer for another company and there are some really good takeaways. I love stories because I can feel that we can all learn from mistakes others have made, and I think this story will help some of you out there avoid a disaster VoIP install like this company went through. I hope you never have to deal with a Chief of Police looking over your shoulder as you’re trying to fix a VoIP problem like James did in this story. I’ve also got another free, cool gift for you guys. Over the years, I’ve noticed that voice is not exactly IT professionals’ forte. So, to help you out, I spent some time and put together a chart comparing the 3 main types of voice access: PRI, SIP and Analog Lines. This chart shows the differences between these three voice access when it comes to things like failover, phone numbers, access methods and “best fit.” We will send you a free copy of this voice comparison guide, if you text the word “voiceguide” to the number 44222. Again, just text the word “voiceguide” to the number 44222 and we’ll send you a free soft copy of this chart that literally took me several hours to compile. Ok, so let’s get to the interview.

Alright. Hey, thanks for joining us, James.

James: Thanks for having me, Mike.

Mike: Can you take a minute and tell us a little bit about yourself both personally and professionally?

James: Sure. I’ve been in telecom for about six years. Started in 2010 working for a small ILEC in upstate New York. That was my first foray into telecommunications. It was a part-time helpdesk job repairing DSL modems over the phone with elderly customers. From there, it, sort of, really taken off in an unexpected way and I just moved to Los Angeles for a position with Velis4 in March of 2016.

Mike: Oh, awesome. How do you like LA so far?

James: I love it. I’ve always wanted to live in LA since the first time I’ve visited in 2009. This was finally my opportunity to get here with a job lined up ahead of time.

Mike: Very cool. It’s a good perspective. I’ve lived in Southern California my whole life, so I’m always curious of what people’s perspective is coming in, what they think of it. Because, you ask that question and you’ll get all kinds of different answers of what people think of LA – some people love it, some people don’t, so I’m always curious. What do you really like about it?

James: I really, really love the weather. I hated the winter on the East Coast, so I’m looking forward to a warm winter this year. I’m also really enjoying the beaches and how close they are to everything. The beach was always a three-hour drive away back on the East Coast.

Mike: Yeah. It’s definitely a scene in the summertime here in Southern California. You go to the beach – it is a whole lifestyle, so that’s great. Yeah. I’ll have to check in with you halfway through the summer to see how you’re enjoying it.

James: Yeah, absolutely. I’m looking forward to it.

Mike: Awesome. Okay. Today, James is going to tell us a little story. He’s going to take us back to a time when he experienced a SIP implementation that was a large SIP implementation and it didn’t go so well. He’s going to, kind of, walk us through how they figured out what was going on, and how they solved the issue, and got the customer to the point where SIP was working very well. I thought it would be a great story to share with all of our listeners since SIP is one of those things, I think, every IT professional is probably a little bit scared of if they’re implementing it for the first time, and they’re worried about exactly what happened on this implementation.

So, I’ll turn it over to you, James, and let you, kind of, take it from there.

James: Sure, thanks. This particular story goes back to 2013. It was a hosted implementation in a large municipality building for a township in New Jersey. It was a really large account. It billed around $15,000 a month and there was potential for it to grow from that initial implementation.

Mike: Wow. That’s pretty big.

James: Yeah, it was a big deal. It was high-profile for the company that I was with at the time because they had relationships with the city where the company was based, so lots of ins, and lots of relationships there that were really important to make sure that we maintain that and did a good job.

Mike: Sure.

James: So, what happened during the initial implementation… We’ve got everything turned up and running through the normal process. They tested, the customer was happy. They were looking forward to realizing their cost savings over their traditional PBX with the hosted environment as well as taking advantage of some of the extra technical features it offered.

About a week after the initial turn-up, we started getting a lot of support tickets and a lot of emails from the customer saying they were having non-stop issues with phones rebooting, calls dropping, they couldn’t use the phones reliably for conferences. They actually ran a police dispatch department off of the hosted phones and the dispatchers were trying to take calls from the officers in the field, and phones were rebooting, dropping calls on them, and it was affecting dispatch operations.

Mike: Oh, man. That’s huge. That’s scary.

James: Yeah. So, you know, you have a dispatcher there that’s trying to deal with, potentially, a 911 situation, or an emergency, or a life-threatening situation, and their phone is rebooting on them. It was completely unacceptable.

Mike: So, was the company you were at, were they supplying them the phones as well in terms of, like, the hosted platform as well as the SIP service or were you guys just doing the SIP service and someone else was doing the actual hardware?

James: We were doing the hardware and the SIP as well.

Mike: Okay, great.

James: So, we had deployed the phones out as well, as the SIP side of it, and the call flow was all with my company.

Mike: Okay. Now, was this at Velis4 or was this a different company?

James: This was a different company.

Mike: Okay. Cool. We can trash them and everything, and it won’t matter – just kidding. So, it sounds like they’re having a lot of issues – continue please.

James: Sure. At the time, at the company that I was at, I was the lead helpdesk technician for Tier 2/Tier 3 support. Eventually, the tickets were escalated to me and I was asked to review it by the manager. I took a look at everything through our monitoring systems, our tools, our traditional way of troubleshooting. We weren’t able to really find anything other than the fact that we saw the disconnects and the reboots happening from the customer side as far as our SBC was concerned. Our SBC was always seeing disconnections in the customer network, which led us to believe that there was something happening specifically on-site and it wasn’t anything upstream that was disconnecting the calls or causing the phones to reboot.

Once we had reached that conclusion, we had decided that it was time to go on-site. Before we had gone on-site, we were getting absolutely scathing emails from the customer. I can remember getting an email from the customer. It was written in all caps with red font, bold, red font, and it, basically, said, “If you don’t fix these issues in a timely fashion, we’re going to throw the phones out onto the highway and let them get run over by the traffic.

Mike: Oh, wow. That’s pretty bad. How long had it been going on at that point? I know you said the issues started to happen about a week into it. How long had it been to this point, roughly?

James: It had been, probably, about a month.

Mike: Oh, man.

James: So, they were really starting to get frustrated at that point. After that email, I got the attention of high-level management within the company and I said, “We really need to do something about this. It needs to be dealt with immediately.” So, myself and one of the backend BroadSoft engineers that we had on staff took a trip down to the office. It was about an hour drive from our location, so we went down there. We were, basically, observing. We wanted to see some phones reboot, we wanted to see calls dropped ourselves.

During that initial visit, we were unable to witness anything, which tends to happen, as I’m sure many people who are listening to this can relate. Sometimes, when you go to observe things for yourself, you just don’t see it, and then you’re led to believe that the customer is crazy.

Mike: Right. I think I have that issue sometimes when my wife says that there’s something wrong with her computer. I’m the IT guy in my house, so I can relate to that.

James: Yeah. It just tends to happen that way, unfortunately, sometimes. So, we left after that initial visit not really having gotten anywhere. That was, of course, with the anxiety of having to meet the customer, which included the Police Chief who was really not very happy with us, understandably so.

Mike: Wow.

James: So, we got over those hurdles…

Mike: Sorry to interrupt, but I don’t think too many people can say that they had the Police Chief meet them on-site for a troubleshooting session.

James: Yeah. It wasn’t fun. It was full of anxiety.

Mike: Jeez.

James: So, we left that first, initial site visit, not really convinced that it wasn’t their issue at that point. We were still suspecting things like inside wiring and that sort of thing, but really had no way to prove it. We didn’t have the right tools. We’re basically just technicians that troubleshoot with our computers. We didn’t have the right tools to troubleshoot inside wiring or anything like that.

Mike: Okay.

James: About a week goes by after that, and we continued to get the nasty phone calls and emails from the customer, at that point, they were communicating directly with me, and we were led to another site visit. This time, the Chief Information Officer for the company came on-site with us. He had been briefed on the situation. He wasn’t so sure, himself, that it was our issue, at that time, and came on-site with us.

We ended up tracing every cable in their network room back to the patch panel to rule out any issues with cross cables, or voltage, or anything along those lines that would cause phones to reboot while they’re on a call.

Mike: How did you guys do that? Because I’m just thinking everything is running through, like, a conduit or something. When you say you “traced” it, did you test each one individually from both sides? How did you do that?

James: Yeah. The rack in this building was a mess. It was basically a rat’s nest. So, we traced every cable from the switch out to the patch panel core and then toned it out from there throughout the entire building.

Mike: Okay.

James: We, basically, had to make sure that we knew where every cable was going and that it was a good cable. What many people don’t know, the Polycom phones won’t actually reboot when it’s on a call. When it’s passing RTP traffic, it knows not to reboot. The only way that it will reboot during a call is if there is a power issue to it, so we suspected something with their inside wiring or power fluctuations that were the root cause.

Mike: Were they doing power over Ethernet or are the phones powered individually?

James: They were power over Ethernet through the LAN switch, so that’s what led us to tracing every cable and toning it out to make sure that there weren’t any issues.

Mike: That makes sense.

James: That was another whole day at the site and we, basically, weren’t able to find any power issues after we had done all that work. Now, at this point, they’re still having issues. It’s probably about two months into it and we’re, basically, scratching our heads. I’m standing there with the leadership of the network team for our company and they’re scratching their heads just as much as I was.

During that initial visit where we did all the cable tracing, we got access to the switches from their IT person on site and we began going through the LAN switches. They were Dell PowerConnect switches that we really had not had any interaction with, from our perspective, as implementation goes. We never really used them at any other sites and we didn’t know much about them at all.

So, after we got access to the switches, we began running through the logs in the switches, and we noticed that ports were shutting down randomly on these switches. It wasn’t just a quick blip of the port, it was a full shutdown, power was cut to the port as well as connectivity over the Ethernet. Once we found that, we knew that we had hit the nail on the head and finally discovered what was causing the phones to reboot.

Mike: Awesome. Tell us a little bit about what you guys found.

James: We were in the logging on the Dell switches and it was showing us all these ports shutting down, so we got on the phone with Dell support. Thankfully, the switches had support on them because they weren’t managed by us, they were the customer’s. At this point, we were just trying to do whatever we could to help them to fix this.

Mike: Sure.

James: So, they had support on the switches and we got a ticket open with them. Basically, Dell support said, “Your firmware version is corrupt. You have to upgrade it and your issue will be resolved.” So, we listened to them, we rolled out the firmware upgrade and it resolved all those issues and all that heartache.

Mike: Wow.

James: It ended up coming down to the firmware on the switches and, basically, due diligence not being done on either side of the implementation.

Mike: So, the Police Chief and all of his staff withdrew their weapons? Yeah. Oh, man. So, at that point, how did you address that with the client? Did the client say, “Hey, sorry, this whole thing was on us and sorry for the bold, red text on the email?” or did they say, like, “Hey, from our perspective, you guys should have known that upfront”? What takeaways did you get from it overall?

James: So, from the client’s perspective, they never apologized because, as far as they were concerned, it was our technology, our implementation, and we were responsible for knowing that. In one sense, I would agree with that, but, at the same token, they had IT personnel on staff that manage those switches, and you could really put some of the blame on them for not being willing to look into the switches or knowing what was happening inside of them.

Mike: Right.

James: But, still, as the account service provider, we should have either said, “We’re not going to use these switches because we don’t know anything about them” and offered to put our managed switches in there, or made it clear and set the expectation upfront to the customer that it’s their responsibility for the switches and we can’t touch them at all even though our phones connect to them.

Mike: Correct. I’m just thinking, so these ports shutting down, was that happening on all their network traffic or just VoIP traffic?

James: Just the VoIP traffic. They actually had dual drops at this location, so their PCs weren’t affected by this.

Mike: Okay.

James: So, from an end-user perspective, it really looked like it was just something wrong with the phones.

Mike: Okay. Were these new switches that they had rolled out just for the second data drops at all the desks?

James: They had added additional switches during the implementation, yes, the customer did.

Mike: Okay, got it. So, from an end-user perspective since our audiences are IT professionals, what could they do, if they’re looking to implement SIP, to troubleshoot their switches? Is there anything, any takeaway that you have, like, “Hey, when you’re up and running SIP, one thing you want to do with your switches is…” – what?

James: One thing you would one to always do is make sure, first of all, that the switches have the newest version of firmware on them. I would always review their release notes on the firmware version for your switch that you’re running at your site. Make sure that it can handle the power load that the phones require and that it’s going to be able to provide you with the necessary logging that you need should you have to troubleshoot.

Mike: Got it. Also, be aware of, in terms of signing up for the service contract on the switches as well, so, at the end of the day, you can call and have some support there ready to go.

James: Yeah and always ask your sales person that you’re working with at the hosted provider if they provide support on LAN switches, because many times it’s, sort of, a gray area. It’s not specified in the contract, it’s not ever talked about upfront, and it, sort of, gets forgotten when, really, that’s part of the heart and soul of the SIP implementation.

Mike: Yeah, I think that’s huge. It’s like there are so many moving parts with SIP and there are so many different stories that can be told of little glitches that caused huge problems. I think that that was a really cool story because, like I said, so many stories can be told about issues with SIP. Really, it’s a fantastic technology and it works pretty well once everything is squared away, but just that initial implementation can sometimes be rough if the proper measures have not been taken upfront.

Moving forward, did that customer continue the service and everything was pretty much smooth sailing after that?

James: Yeah. After we had gotten that issue sorted out with their Dell switches, everything ran smoothly, they stopped opening tickets. We continued to follow up with them, but everything was going smoothly – they had no further issues.

Mike: About how many users did they have?

James: There was approximately a 120 phones in that single location.

Mike: Wow. What’s crazy about that is you told me on the pre-call that this was a hosted VoIP application over third-party bandwidth, correct? I think you said they were using business cable there for their internet connectivity?

James: Yeah, they were. They ran it over coax.

Mike: So, they had over a hundred users on hosted VoIP over third-party bandwidth, and not just third-party bandwidth, not fiber, but business-class cable, and they didn’t have any issues beyond that. The issue that they had was internal, on their switches.

That’s pretty cool because, I think, selling hosted as well… You know, we deal a lot with providers and you have to have dedicated bandwidth in order to have good service. I’m sure that was probably going through their mind as they were having those issues, right? They were thinking, “Jeez, is this a result of us not having dedicated circuit and us trying to run this over business-class cable?” Was that kind of conversation going on as well?

James: Yeah. That was certainly part of the early on conversation when we were initially troubleshooting. Fortunately, it’s pretty easy to rule out your circuit or not when it comes to RTP and SIP. You can have that ruled out in a couple of hours along with some consistent monitoring over a period of time.

Mike: That’s awesome. Well, cool story. Thanks for sharing. I think there are some good takeaways in there for everybody.

James: Yeah, my pleasure.

Mike: Alright. Now, I’d like to ask you, kind of, a fun question. Tell us about the coolest, or most interesting, or funniest, or strangest thing you’ve ever experienced at work.

James: Yeah, absolutely. This is, kind of, nerdy of me, but when I first started back in upstate New York in about six years ago, the first time I ever saw a central office, I was just absolutely fascinated by it, because that was the first time I had ever seen all of the interconnections in how the utility poles and their connections come down to the ground and get out to houses and businesses.

I just thought that that was the most fascinating thing. I had never truly realized how intricate and detailed it was until that point. It was just, sort of, like, a wow moment for me, because I had never seen it and I had no idea how it worked. It’s just something that, you know, you always see when you’re driving by on the street or something like that, and you never really think about what goes into it, so I was really fascinated by that.

Mike: That’s cool. How much engineering training or education had you had prior to that visit?

James: Prior to that, I had just been on the job for, maybe, three months part-time taking phone calls and, maybe, logging in to some central office equipment, but not actually seeing any of it in the flesh. It was all really new to me.

Mike: What stands out to me is that, hey, everyone, this is a sign that you are a future engineer for a service provider. If you ever go on a visit or a tour of a central office and it is fascinating to you, note to self, this is going to be your profession.

This is awesome because, for me, when I went on my first of a data center, I remember… I was working for XO Communications and I remember it was cool looking, but 90% of the stuff they were talking about, I had no idea what they were saying, so as much as I was really trying to pay attention, it was really hard to get what was taking place. I was like, “Man, this thing looks really complicated and I don’t want to touch anything,” but that’s cool that you, kind of, just lit up the first time you walked into one of those suckers.

James: Yeah. I thought it was absolutely fascinating. It really was.

Mike: Awesome. Well, cool. Tell us a little bit about Velis4. So, we’ve heard from you, we’ve heard the cool story and everything, but this is the part where I like to put the engineers on the spot and make them almost be marketing people for a second, usually, kind of, read from the script that the marketing person gave you to say on the podcast. Tell us a little bit about Velis4 and what you guys have going on that’s fun these days or interesting.

James: Sure. It would be my pleasure. So, Velis4 focuses on the needs of businesses looking to consolidate, improve, and streamline communications and technology by leveraging cloud. Velis4 accomplishes with advance features such as unified communications, using best in class platforms. We actually have four platforms that we can offer a unique product set from, including Microsoft, Skype for Business, BroadSoft, NetSapien, and Sonus.

Customer benefits include advanced call center capabilities, leveraging mobile solutions, embracing softphones, and the ability to integrate with other technology including CRMs and other customer applications. We can assist in advance network design and implementation to support any type of cloud infrastructure and assist in monitoring that network as well. With an eye toward the cloud, we’re also able to offer remote desktop capabilities for any organization for true business continuity and security.

Mike: Fantastic. I know you guys offer SIP, you guys also offer hosted VoIP…

James: That’s correct.

Mike: Then, in terms of like a product set that customers would recognize… We talked a little bit about call center/contact center. That sounds like a product set for you guys as well.

James: Yes. We have call center capabilities on the BroadSoft and the NetSapien switch.

Mike: Fantastic. Any other products or services that I missed there?

James: Oh, we actually can also offer dedicated bandwidth and circuits as well.

Mike: Okay, great. Very cool. So, from your perspective, what’s your day-to-day like? What do you enjoy most about your day-to-day and, kind of, take us to what you enjoy about your current position?

James: Sure. My current position, I am more leading the NOC here. I am able to, unlike in previous positions that I’ve had, I’m able to take ownership of the operations for the NOC and have the power to make decisions and changes as we need to without being tied down by other management restrictions. So, it’s really a great position to take ownership and accountability for everything that needs to be done here to support our customers. That’s really what I enjoy doing on a day-to-day basis – making sure that they have the best possible service experience with us.

Mike: Very cool. That’s awesome. Well, cool. It’s been fun having you on the show, James. I think you gave us some great info and some fun info as well, and we appreciate having you.

James: My pleasure. Thanks, Mike.

Mike: Alright. Have a good day and enjoy that summer.

James: You too. I will. Thanks.

Mike: Okay, bye.

James: Bye.

Mike: Alright. So, what did you think about James? Wasn’t he great? Man, that story brought back a lot of pain to me. Being in telecom for sixteen years, boy, having issues that can’t be resolved for days on end… Wasn’t it funny that the customer is emailing him in all-caps and red ink, and the Chief of Police is there, the CIO is there. I mean, gosh, that was a painful story to have to hear, but I think there are some awesome takeaways from it, so I hope you got something from that. I know I did.

Before I go, I wanted to remind you again about the free gift. Again, it’s a chart a chart comparing the 3 main types of voice access: PRI, SIP and Analog Lines. This took me quite a while to create but I know you’ll get some good information from it. The chart shows the differences between these three voice access, when it comes to things like failover, phone numbers, access methods and “best fit.”

We will send you a free copy of this voice comparison guide, if you text the word “voiceguide” to the number 44222. Again, just text the word “voiceguide” to the number 44222 and we’ll send you a free soft copy of this chart that literally took me several hours to compile.

Alright. Catch you next time!