Preface

I’m familiar enough with DynamoDB to know (and have experienced) a number of, shall we say, peculiarities inherent to Amazon’s (let’s be honest, incomplete) implemention. I am not familiar enough to say I know DynamoDB, but I have enough data retrieval experience that I get the general concepts. I’m going to guess I’m not the only dev out there in this position.

A Basic Query?

I recently jumped into an existing codebase to do some new product work. One aspect of that work was to retrieve a record by its id from DynamoDB. id had been designated as the primary key:

"KeySchema": [
    {
      "AttributeName": "id",
      "KeyType": "HASH"
    }
]

Seems straight forward enough. Somehow, this codebase hadn’t yet done this most basic type of query. All pre-existing queries operated against global secondary indexes.

I began as any seasoned developer would - I copied the existing code:

table = await dynamodb.Table('table-1')
key_condition = Key('global-secondary-index-name').eq(some_id + '|' + some_string)
await table.query(IndexName='global-secondary-index-name', Limit=1, KeyConditionExpression=key_condition, ScanIndexForward=False)

Should be easy enough to adapt, right? Remember, I needed to query against the primary key, which is id. I modified as follows:

table = await dynamodb.Table('table-2')
key_condition = Key('id').eq(some_id)
await table.query(IndexName='id', Limit=1, KeyConditionExpression=key_condition, ScanIndexForward=False)

(If you already know the problem above just keep it to yourself for now, k? Don’t ruin my story.)

OMG My Index Is Wrong

I wrote ¹ a test and ran it as a formality. Of course it’ll pass, right? Wrong. Here’s the error message I got:

botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid length for parameter IndexName, value: 2, valid range: 3-inf

Uhm, huh? My index name is too short? But…this index exists on the table already. It’s there, Dynamo. You agreed to make it already. It’s a little late for this complaint, no?

Withholding Information is Lying

I won’t drag this out any longer. Before enumerating my complaints, here’s the correct code:

table = await dynamodb.Table('table-2')
key_condition = Key('id').eq(some_id)
await table.query(KeyConditionExpression=key_condition)

Notice that IndexName is no longer present in the call to table.query ². IndexName is only for querying Global Secondary Indexes. The problem with my original attempt is obvious after knowing this. But without knowing this non-obvious implementation detail of boto, one might spend a non-insignificant amount of time trying to rename the id primary index and continue to receive unrelated rabbit-hole error messages that only serve to take one further away from the actual problem.

If one were to come at this problem the other way around - the existing code queried the primary index and the new ask was to query a global secondary index, I’d imagine a much more direct debugging session.

And now, at long last, my sepcific complaints:

1. IndexName isn’t a great name for this parameter. I know GlobalSecondaryIndexName is really verbose, but at least it’s clear.
2. The error message above - Invalid length for parameter IndexName, value: 2, valid range: 3-inf - while not wrong, is also not helpful. It assumes I know #1 above, and, failing that, offers no other real help.

Each One Teach One

I didn’t find a quick answer to my issue by searching. In fact, in a twist none of you saw coming, I was the one referred to above going down rabbit holes! Shock! Awe!

My hope is that this post ends up in some search algorithm and saves folks from falling down the same rabbit hole.

Until next time, take care of each other, and I’ll see you on the internet.

~ jonathan, backend team

1. And by ‘wrote’ I mean copied an existing test and tweaked it. We’re not in the typing business here. (Well, here on this blog we are, but not, you know, for our actual jobs.) ↩

2. The other params, Limit and ScanIndexForward don’t really play a role in this example. I should have left them out entirely but also wanted to show the original code and be “true” to the story. ↩

Tagged with: #dynamodb, #python, #boto3

---