seeing continuous HTTP 500 errors on HeadObject calls for ~3 hours with no recovery
▎ Details:
▎ - Endpoint: https://t3.storage.dev
▎ - Bucket:
▎ - Affected paths:
▎ - Operation: HeadObject via boto3/s3fs
▎ - Error: ClientError: An error occurred (500) when calling the HeadObject operation (reached max retries: 4): Internal Server Error
▎ - Started: ~20:03 UTC, April 5, 2026
▎ What we've ruled out:
▎ - Same bucket and paths work fine from AWS us-west-2
▎ - Fails from multiple independent non-AWS networks (bare metal datacenters and DigitalOcean)
▎ - Rotated credentials, same 500 with new key
▎ - Other paths in the same bucket accessible from AWS
▎ - Status page shows all operational
▎ - Endpoint: https://t3.storage.dev
▎ - Bucket:
meshy-learningmeshy-learning▎ - Affected paths:
checkpoints/meshylatv_avocado/VAE_20251019_d0e667e7/checkpoints/meshylatv_avocado/VAE_20251019_d0e667e7/ and checkpoints/meshylatv_avocado/GSHead_GANS_20251109_38541c8e/checkpoints/meshylatv_avocado/GSHead_GANS_20251109_38541c8e/▎ - Operation: HeadObject via boto3/s3fs
▎ - Error: ClientError: An error occurred (500) when calling the HeadObject operation (reached max retries: 4): Internal Server Error
▎ - Started: ~20:03 UTC, April 5, 2026
▎ What we've ruled out:
▎ - Same bucket and paths work fine from AWS us-west-2
▎ - Fails from multiple independent non-AWS networks (bare metal datacenters and DigitalOcean)
▎ - Rotated credentials, same 500 with new key
▎ - Other paths in the same bucket accessible from AWS
▎ - Status page shows all operational
Tigris
Tigris is globally distributed, S3-compatible object storage with zero egress fees. Scales infinitely, works across any cloud, and provides instant access to your data worldwide.
Solution
To help others find answers, you can mark your question as solved via
Right click solution message -> Apps -> ✅ Mark SolutionRight click solution message -> Apps -> ✅ Mark Solution
