ECS Image Resolution and Pull Through Cache Solution

July 10, 2025

I ran into an interesting issue with ECS tasks that taught me about Docker image resolution and mutable tags.

The Problem

When running an ECS task, I pointed to a specific image tag but discovered that ECS actually resolves image tags to their SHA256 digests. Even though I specified something like my-repo:v1.0.0, ECS was internally using the SHA that this tag resolved to at deployment time.

The issue became apparent when my private image syncer (dregsy) started overwriting images. I didn’t realize that Docker tags are mutable - the same tag can point to different SHAs over time. What really caught me off guard was that even versioned tags like v1.0.0 were mutable, not just latest. I had assumed that specific version tags would be immutable since that’s typically the best practice for image builds. So when dregsy synced a newer version of the image with the same tag, it overwrote the image that my ECS task was expecting to use.

The Solution: Pull Through Cache

The solution was to implement a pull through cache. Here’s why this works:

SHA Resolution: When the pull through cache pulls an image, it resolves the tag to its specific SHA256 digest
Immutable Storage: The cache stores the image by its SHA, which is immutable
Consistent Retrieval: Even if the original tag’s SHA association changes, the cached image remains unchanged

By pointing my ECS tasks to use the pull through cache, they will resolve the SHA and pull that specific digest down. This cached version will never be overwritten, even if the original tag in the source registry later resolves to a different SHA. This means my ECS tasks get consistent, predictable image versions regardless of what happens to the mutable tags in the source registry.

Key Learnings

Docker tags are mutable - the same tag can point to different image SHAs over time
ECS resolves tags to SHAs - it doesn’t just store the tag reference
Pull through caches provide stability - they cache the resolved SHA, not just the tag
Image syncers can cause unexpected overwrites - especially with mutable tags

This experience highlighted the importance of understanding how container orchestration platforms handle image resolution and the benefits of using pull through caches for production workloads.