What is the easiest way to implement offset and limit in AWS DynamoDB?

I am currently working on DynamoDB and I saw there is no inbuilt function to put offset and limit like SQL queries. The only thing I have seen is get the lastEvaluatedKey and pass as exclusiveStartKey. But my problem is according to our scenarios we can't get the lastEvaluatedKey and populate because we are sorting data and parallel-stream the data.

So there will be issues and hard points. I just need to know what is the clean way or are there any best way we can pass the offset and limit and get the data without iterating over all the data. Because right now even though there is offset and limit put to bound-iterator it is inside get and iterating all the data in DynamoDB which consumes lots of read capacities even though we don't need the other data.

Map<String, AttributeValue> valueMap = new HashMap<>();
valueMap.put(":v1", new AttributeValue().withS(id));

Map<String, String> nameMap = new HashMap<>();
nameMap.put("#PK", "PK");

DynamoDBQueryExpression<TestClass> expression = new DynamoDBQueryExpression<TestClass>()
.withKeyConditionExpression("#PK = :v1")
.withExpressionAttributeNames(nameMap)
.withExpressionAttributeValues(valueMap)
.withConsistentRead(false);

PaginatedQueryList<TestClass> testList = dynamoDBMapper.query(TestClass.class, expression);
Iterator<TestClass> iterator = IteratorUtils.boundedIterator(testList.iterator(), offset, limit);
return IteratorUtils.toList(iterator);

What will be the best way to handle this issue?


A Query request has a Limit option just like you wanted. As for the offset, you have the ExclusiveStartKey option, which says at which sort key inside the long partition you want to start. Although usually one pages through a long partition by setting ExclusiveStartKey to the LastEvaluatedKey of the previous page, you don't strictly need to do this, and can actually pass any existing or even non-existing item, and the query will start after that key (that's the meaning of the word exclusive, i.e., it excludes).

But when you said "offset", you probably meant a numeric offset - e.g., start at the 1000th item in the partition. Unfortunately, this is not supported by DynamoDB. You can approximate it if your sort key (or LSI key) is a numeric offset if the item (only practical if you only append to the partition...) or using some additional data structures, but it's not supported by DynamoDB itself.