Make elasticsearch only return certain fields?

Solution 1:

Yep, Use a better option source filter. If you're searching with JSON it'll look something like this:

{
    "_source": ["user", "message", ...],
    "query": ...,
    "size": ...
}

In ES 2.4 and earlier, you could also use the fields option to the search API:

{
    "fields": ["user", "message", ...],
    "query": ...,
    "size": ...
}

This is deprecated in ES 5+. And source filters are more powerful anyway!

Solution 2:

I found the docs for the get api to be helpful - especially the two sections, Source filtering and Fields: https://www.elastic.co/guide/en/elasticsearch/reference/7.3/docs-get.html#get-source-filtering

They state about source filtering:

If you only need one or two fields from the complete _source, you can use the _source_include & _source_exclude parameters to include or filter out that parts you need. This can be especially helpful with large documents where partial retrieval can save on network overhead

Which fitted my use case perfectly. I ended up simply filtering the source like so (using the shorthand):

{
    "_source": ["field_x", ..., "field_y"],
    "query": {      
        ...
    }
}

FYI, they state in the docs about the fields parameter:

The get operation allows specifying a set of stored fields that will be returned by passing the fields parameter.

It seems to cater for fields that have been specifically stored, where it places each field in an array. If the specified fields haven't been stored it will fetch each one from the _source, which could result in 'slower' retrievals. I also had trouble trying to get it to return fields of type object.

So in summary, you have two options, either though source filtering or [stored] fields.

Solution 3:

For the ES versions 5.X and above you can a ES query something like this:

    GET /.../...
    {
      "_source": {
        "includes": [ "FIELD1", "FIELD2", "FIELD3" ... " ]
      },
      .
      .
      .
      .
    }

Solution 4:

In Elasticsearch 5.x the above mentioned approach is deprecated. You can use the _source approach, but but in certain situations it can make sense to store a field. For instance, if you have a document with a title, a date, and a very large content field, you may want to retrieve just the title and the date without having to extract those fields from a large _source field:

In this case, you'd use:

{  
   "size": $INT_NUM_OF_DOCS_TO_RETURN,
   "stored_fields":[  
      "doc.headline",
      "doc.text",
      "doc.timestamp_utc"
   ],
   "query":{  
      "bool":{  
         "must":{  
            "term":{  
               "doc.topic":"news_on_things"
            }
         },
         "filter":{  
            "range":{  
               "doc.timestamp_utc":{  
                  "gte":1451606400000,
                  "lt":1483228800000,
                  "format":"epoch_millis"
               }
            }
         }
      }
   },
   "aggs":{  

   }
}

See the documentation on how to index stored fields. Always happy for an Upvote!