June 16th, 2020 6:23pm PST
Update on T-Mobile Voice and Text Performance
Every day we see the vital role technology plays in keeping us connected, and we know T-Mobile customers rely on our network to ensure they have connections with family, loved ones and service providers. This is a responsibility my team takes very seriously and is our highest priority. Yesterday, we didn't meet our own bar for excellence.
Many of our customers experienced a voice and text issue yesterday, specifically with VoLTE (Voice over LTE) calling. My team took immediate action -- hundreds of our engineers worked tirelessly alongside vendors and partners throughout the day to resolve the issue starting the minute we were aware of it. Data connections continued to work, as did our non-VoLTE calling for many customers and services like FaceTime, iMessage, Google Meet, Google Duo, Zoom, Skype and others allowed our customers to stay in touch. Additionally, many customers were able to use circuit-switched voice connections and customers on the Sprint network were unaffected. VoLTE and text in all regions were fully recovered by 10 p.m. PDT last night. I'm happy to say the network is fully operational… and we're working day in and day out to keep it that way.
Our engineers worked through the night to understand the root cause of yesterday's issues, address it and prevent it from happening again. The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we've worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don't affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.
We have worked with our IMS (IP Multimedia Subsystem) and IP vendors to add permanent additional safeguards to prevent this from happening again and we’re continuing to work on determining the cause of the initial overload failure.
So, I want to personally apologize for any inconvenience that we created yesterday and thank you for your patience as we worked through the situation toward resolution.
Neville Ray
T-Mobile President of Technology
Update - June 15, 2020 10:03pm PST
These issues are now resolved. We again apologize for any inconvenience and thank you for your patience.
June 15th, 2020 8:45pm PST
Starting just after 12 pm ET and continuing throughout the day, T-Mobile has been experiencing a voice and text issue that has intermittently impacted customers in markets across the U.S. We are recovering from this now but it may still take several more hours before customer calling and texting is fully recovered. Neville Ray has shared updates throughout the day but I wanted to share the latest on what we know and what we're doing to address it. This is an IP traffic related issue that has created significant capacity issues in the network core throughout the day. Data services have been working throughout the day and customers have been using services like FaceTime, iMessage, Google Meet, Google Duo, Zoom, Skype and others to connect.
I can assure you that we have hundreds of our engineers and vendor partner staff working to resolve this issue and our team will be working through the night as needed to get the network fully operational.
Mike Sievert
Director ejecutivo