What is the easiest way to implement offset and limit in AWS DynamoDB?
I am currently working on DynamoDB and I saw there is no inbuilt function to put offset and limit like SQL queries. The only thing I have seen is get the lastEvaluatedKey
and pass as exclusiveStartKey
. But my problem is according to our scenarios we can't get the lastEvaluatedKey
and populate because we are sorting
data and parallel-stream
the data.
So there will be issues and hard points. I just need to know what is the clean way or are there any best way we can pass the offset and limit and get the data without iterating over all the data. Because right now even though there is offset and limit put to bound-iterator
it is inside get and iterating all the data in DynamoDB which consumes lots of read capacities even though we don't need the other data.
Map<String, AttributeValue> valueMap = new HashMap<>();
valueMap.put(":v1", new AttributeValue().withS(id));
Map<String, String> nameMap = new HashMap<>();
nameMap.put("#PK", "PK");
DynamoDBQueryExpression<TestClass> expression = new DynamoDBQueryExpression<TestClass>()
.withKeyConditionExpression("#PK = :v1")
.withExpressionAttributeNames(nameMap)
.withExpressionAttributeValues(valueMap)
.withConsistentRead(false);
PaginatedQueryList<TestClass> testList = dynamoDBMapper.query(TestClass.class, expression);
Iterator<TestClass> iterator = IteratorUtils.boundedIterator(testList.iterator(), offset, limit);
return IteratorUtils.toList(iterator);
What will be the best way to handle this issue?
A Query request has a Limit
option just like you wanted. As for the offset, you have the ExclusiveStartKey
option, which says at which sort key inside the long partition you want to start. Although usually one pages through a long partition by setting ExclusiveStartKey
to the LastEvaluatedKey
of the previous page, you don't strictly need to do this, and can actually pass any existing or even non-existing item, and the query will start after that key (that's the meaning of the word exclusive, i.e., it excludes).
But when you said "offset", you probably meant a numeric offset - e.g., start at the 1000th item in the partition. Unfortunately, this is not supported by DynamoDB. You can approximate it if your sort key (or LSI key) is a numeric offset if the item (only practical if you only append to the partition...) or using some additional data structures, but it's not supported by DynamoDB itself.