Aggregation + sorting +pagination in elastic search

I need to do an aggregation + sorting + pagination in one of the indexes.

I learned about internal functionality of Elastic search,

I have 5 total shards, it will sort the individual shards and fetch the result, by default each shard will return in 10 records. Then the 50 records are sorted again and it will fetch the top 10 record since by default size is 10.

ouput:

The aggregated results are returned in separate field named as "aggregations".In order to do pagination in this aggregated data,size and from are not working.

So tired of doing termBuilder.size(500), now the logic was differs as per this link (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html)

It leads to inaccuracy of data.

Can any one suggest me how to deal with aggregation + pagination.

Solution 1:

In elasticsearch, it's not possible to paginate an aggregation. The query will not give accurate results if size is specified. So, the only way to do sorting and pagination is to give size 0 and return all the documents and then, get the required results by accumulating all the results in a list for further operation.

Solution 2:

I think Composite Aggregation might solve your problem, as it allows pagination within aggregated results.

Please refer to this doc

Solution 3:

ElasticSearch supports Bucket Sort Aggregation in v6.1 and later. It allows "sort", "size" and "from" parameters within aggregated results.

Please refer to this doc

Solution 4:

Yes it is possible pagination + sorting + searching elasticsearch Open link. Elasticsearch supports Bucket Sort Aggregation in in v6.X and later. This bucket_sort uses all records in terms/date_histogram bucket and apply over that. So for this case we have to keep bucket size big enough or more than bucket records so that it keep all possible records in bucket. Example as follows...

{
    "aggs": {
        "aggs1": {
            "terms": {
                "field": "field_name.keyword",
                // We can do sort here also
                "size": 1000000  // Keep this size big integer. This keep all possible result in bucket
            },
            "aggs": {
                "bucket_sort": {
                    "bucket_sort": {                                 
                        "sort": [{
                            "_key": {
                                "order": "asc"
                            }
                        }],
                        // This "from" and "size" use above terms bucket size. it applies over available bucket data [This one give actual result]
                        // Bellow is standard pagination as we do
                        "from": 0,
                        "size": 10
                    }
                }
            }
        }
    },
    "size": 0
}