High Latency and Packet loss

Yeah, I don't hold much hope that it will. I think you are correct something has broken icmp on TMHI but they won't admit it. The tech didn't tell me what gateways were affected but just that the Arcadyan was not ,which I don't buy it. I think it is all of them considering the Arcadyan and the sagemcom are both experiencing it i am sure the Nokia is too.

The odd thing is that the failover to T-Mobile on the Netgate worked very well until a couple of days ago. I have been working with Netgate Support and we can't find anything wrong with the Netgate.

I would like to think it is a temporary issue but given it is pretty consistently 80% packet loss and extreme latency it is more than likely over aggressive throttling. On another conversation a user in LA reported normal behavior without the excessive packet loss and extreme latency. I haven't seen enough responses to see if there is some regional profile that seems to be related but given the other user that chimed in from AZ on another conversation sees the same thing it is hard to say. It is probably some change T-Mobile has made. I doubt T-Mobile will readily communicate on the matter. 

It is interesting to see Speedtest.net run. I think the ping latency reporting by the application might not actually be from the pings but rather from the TCP session establishment. I know from my packet capture the TCP ports used and can see the three way handshake process etc… plus I see the pings that all seem to fail to get a response. Since I am seeing good speeds up and down and low jitter values it makes me a little suspicious about the reporting methods. Using fast.com the results are pretty similar so both tend to agree and report 47-48 ms latency. 

Ok, Well I got the new router and of course it didn't fix the ICMP issue but I did the the Netgate to work by disabling all down detection monitoring (Ping) for T-Mobile as it is the Tier2 secondary ISP so if starlink is down the it fails over to T-Mobile so down detection monitoring for T-Mobile is kind of moot anyway. The concern is that it did work.

Well, that makes sense. Now you have 100% confirmation. Support is just throwing hardware at the problem blindly. At least the solution is still working.

Yep, Not a hardware issue. There in some network issues that they are probably aware of but don't know how to fix and as you said just throwing hardware at it hopping it may fix it. The weird thing is if you run a OOKLA speed test the ping is not to bad like upper 50s or low 60s which is not to bad but it finds the nearest server however if you actually run a ping from the command line 1 in 10 will return a reply and 65% to 95% packet loss. I think the only ones complaining are the Teckie users or gamers so it is not a huge priority for them. It really wouldn't have been an issue for me had I not setup Dual WAN with pfsense because T-Mobile is my backup ISP. 

I ran the Ookia speedtest.net application and took a Wireshark packet capture to see how that works. At the beginning of the session between the client on my machine and the speedtest.net server I was testing against I could see a series of pings and ALL of them were fails as no response to the ICMP packets sent by the ping utility. The two host connect via a TCP session and there are packets targeted to port 8080 from a different port(s). Depending upon the packet type/content the source port would change. Now if Ookia tunnels some ICMP packets between the two hosts or just has an alternate calculation for the latency, which may be, then it can and does report the value. From what I have seen in the capture the volume and rate of the packets from one client to the other are at a rapid rate so I recorded duplicate ACKs and retransmissions etc… It appeared rather chaotic at times but the test did succeed and values were reported. If they do run an MPLS tunnel for some of the session or not I cannot say but it works. Of the 22 ICMP packets sent all failed so not sure how the value was pulled except for the session establishment via the TCP handshake.

My 5G home internet is having packet loss.  After spending well over an hour with t-mobile support, they insisted there is nothing wrong, argued with me that this was normal.  They continue to say speed and latency are fine… however, REFUSE to understand that this is LOSS OF PACKETS and nothing to do with latency or speed.  Latency and speed are fine… but reliability IS NOT.  How does this manifest?  During my work audio/video calls, I see a pause every 30 seconds or so.  When I watch netflix, sporadically bombs out.  When my kids try to play a game, they see sporadic hangs (aka lag).  I run a ping test and clearly see the packet drops and over 10 minutes of ping… 3% packets are lost.  t-Mobile says they can run a ping test for a duration and things are great on their end… yep… thanks for listening to the customer.

Ping statistics for 8.8.8.8: ~10 mins
    Packets: Sent = 629, Received = 607, Lost = 22 (3% loss),
Approximate round trip times in milli-seconds:
    Minimum = 90ms, Maximum = 137ms, Average = 97ms

I guess t-mobile isn't ready to understand that packet loss at this level is unacceptable or want to bother their network team to take a deeper look.  Until they lose a lot more customers, what else can I do?  The painful option to go back to the much more expensive and GB constrained Cricket (at least I can do my real job) or look to (also expensive) Starlink.

Sad... I was optimistic t-mobile was really trying to help the rural community get connected and they deliver a sub-par experience (no static IPs, poor network reliability) at an affordable price.  I guess you get what you pay for.

The latency is rather high so that doesn't help and probably contributes to the packet loss. In effect you would have poor performance and retransmission of packets. Check your cellular metrics for both the 4G and 5G signals and identify the frequencies. If the 5G is n41 and the tower is not too far away it is good to know and understand the metric values. If your RSRP is good be sure to try to improve the RSRQ and SINR values. If you are able to reposition the gateway to improve the radio signal receive quality and the signal to noise ratio then the result should be improved performance as there should be less packet damage and fewer retransmissions. Sure if you can get improved signal receive power that also will help but try to improve the quality of the signal and don't overlook that. Your location may not have the best signal from the tower so that can be a challenge. 

Thanks for the suggestions.  My latency and speeds are absolutely fine.  Here are my 5G metrics:

BAND n41

RSRQ 2

RSRP -93

SINR 22

From my reading on these metrics, this is all in the Excellent range.  There is a systematic loss every 30 seconds or so.  This didn't happen in January the last time I used the service a lot.  I experienced this issue 20+ years ago with a DSL provider and ended up being a bad switch/router somewhere in their network.  Back then, a smaller phone company who had someone who LISTENED to the customer and actually had someone who knew networks.

T-Mobile Support “we pinged you, your speeds are good… sorry nothing more on our end” and then argues with me with their talking points of “speed good, ping worked”.  knowing I am many other are experiencing sub-par RELIABILITY quality for similar reasons is the mark of a failing service.

They are so close to a great service at a great price… but devil's in the details.  They need to have static IPs for their modems, improve network resiliency and raise the bar on their technical customer support.  I am pursuing Starlink and guess will have to make that investment for a quality internet option in a remote location.