Handling partial batch failures with the SQS Lambda Poller

In my previous post, I described how I’d been asked to help a developer who was having some issues. His team was logging all S3 accesses to a central log bucket and then using Athena to analyse the data. They needed to partition this data to allow Athena to process the data more efficiently. However, a small number of the entries they were trying to process seemed to be missing when they used the architecture below:

Now You See Me, Now You Don't - the Mystery of the Vanishing S3 Objects

One of the great things about my role as a consultant at GlobalLogic is that sometimes I’ll be asked to help out on what at first glance can be a simple problem, but as I investigate, I get a chance to uncover some unusual or forgotten features.

Recently, I was working on a project and was asked if I could help solve an issue that had been puzzling a developer. They had deployed a system where some objects uploaded to an S3 bucket seemed to disappear and then reappear; it was time to start digging.