Intermittent 500-2000ms stalls on object storage PUTs, isolated to 66.93.0.25
Hi, we use Tigris via Railway Buckets and have been chasing intermittent PUT stalls since 2026-09-04. I think I've isolated it to one backend address and would appreciate a sanity check.
Bucket
I pinned both addresses from one client, same process, same minutes, alternating between them. 2.2 KB PUTs, 8 per address per minute, 2026-09-10 10:09-10:28 UTC:
Only the destination address differs.
Phase timing on the stalled requests: TCP connect ~2ms, TLS ~9ms, request body fully flushed by ~9ms, then 470-1970ms of silence, then response headers, then a 0.1ms body. So the request lands quickly and the node is slow to answer it.
It's independent of object size (a 0-byte PUT stalls as often as a 64 KB one) and of connection reuse (fresh connections stall at the same rate as pooled ones). The rate varies between 13% and 23%, and
Two questions:
1. Is anything known to be up with
2. Is the address steered per client IP? Ours is
I raised this with Railway who redirected me here. Happy to provide more info if you need it.
Nick
Bucket
freddy-raw-payloads-prod-uwra7jfreddy-raw-payloads-prod-uwra7j, endpoint t3.storageapi.devt3.storageapi.dev.t3.storageapi.devt3.storageapi.dev returns a single A record that differs per client, either 66.93.0.2266.93.0.22 or 66.93.0.2566.93.0.25. Our containers that get .25.25 are slow for as long as they live; the ones that get .22.22 are fine.I pinned both addresses from one client, same process, same minutes, alternating between them. 2.2 KB PUTs, 8 per address per minute, 2026-09-10 10:09-10:28 UTC:
Only the destination address differs.
Phase timing on the stalled requests: TCP connect ~2ms, TLS ~9ms, request body fully flushed by ~9ms, then 470-1970ms of silence, then response headers, then a 0.1ms body. So the request lands quickly and the node is slow to answer it.
It's independent of object size (a 0-byte PUT stalls as often as a 64 KB one) and of connection reuse (fresh connections stall at the same rate as pooled ones). The rate varies between 13% and 23%, and
.25.25 goes clean for five-minute stretches.Two questions:
1. Is anything known to be up with
66.93.0.2566.93.0.25?2. Is the address steered per client IP? Ours is
162.220.232.66162.220.232.66 and it's stable for a container's lifetime, so if we land on a degraded node we can't move off it except by redeploying until Railway hands us different egress.I raised this with Railway who redirected me here. Happy to provide more info if you need it.
Nick
