© 2013-2021 The original author(s).

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Preface

The Spring Data Elasticsearch project applies core Spring concepts to the development of solutions using the Elasticsearch Search Engine. It provides:

  • Templates as a high-level abstraction for storing, searching, sorting documents and building aggregations.

  • Repositories which for example enable the user to express queries by defining interfaces having customized method names (for basic information about repositories see [repositories]).

You will notice similarities to the Spring data solr and mongodb support in the Spring Framework.

1. What’s new

1.1. New in Spring Data Elasticsearch 4.4

  • Upgrade to Elasticsearch 7.17.0.

1.2. New in Spring Data Elasticsearch 4.3

  • Upgrade to Elasticsearch 7.15.2.

  • Allow runtime_fields to be defined in the index mapping.

  • Add native support for range field types by using a range object.

  • Add repository search for nullable or empty properties.

  • Enable custom converters for single fields.

  • Supply a custom Sort.Order providing Elasticsearch specific parameters.

1.3. New in Spring Data Elasticsearch 4.2

  • Upgrade to Elasticsearch 7.10.0.

  • Support for custom routing values

1.4. New in Spring Data Elasticsearch 4.1

  • Uses Spring 5.3.

  • Upgrade to Elasticsearch 7.9.3.

  • Improved API for alias management.

  • Introduction of ReactiveIndexOperations for index management.

  • Index templates support.

  • Support for Geo-shape data with GeoJson.

1.5. New in Spring Data Elasticsearch 4.0

  • Uses Spring 5.2.

  • Upgrade to Elasticsearch 7.6.2.

  • Deprecation of TransportClient usage.

  • Implements most of the mapping-types available for the index mappings.

  • Removal of the Jackson ObjectMapper, now using the MappingElasticsearchConverter

  • Cleanup of the API in the *Operations interfaces, grouping and renaming methods so that they match the Elasticsearch API, deprecating the old methods, aligning with other Spring Data modules.

  • Introduction of SearchHit<T> class to represent a found document together with the relevant result metadata for this document (i.e. sortValues).

  • Introduction of the SearchHits<T> class to represent a whole search result together with the metadata for the complete search result (i.e. max_score).

  • Introduction of SearchPage<T> class to represent a paged result containing a SearchHits<T> instance.

  • Introduction of the GeoDistanceOrder class to be able to create sorting by geographical distance

  • Implementation of Auditing Support

  • Implementation of lifecycle entity callbacks

1.6. New in Spring Data Elasticsearch 3.2

2. Project Metadata

3. Requirements

Requires an installation of Elasticsearch.

3.1. Versions

The following table shows the Elasticsearch versions that are used by Spring Data release trains and version of Spring Data Elasticsearch included in that, as well as the Spring Boot versions referring to that particular Spring Data release train:

Spring Data Release Train Spring Data Elasticsearch Elasticsearch Spring Framework Spring Boot

2022.0 (Raj)

4.4.x

7.17.0

5.3.x

2.7.x

2021.1 (Q)

4.3.x

7.15.2

5.3.x

2.6.x

2021.0 (Pascal)

4.2.x

7.12.0

5.3.x

2.5.x

2020.0 (Ockham)[1]

4.1.x[1]

7.9.3

5.3.2

2.4.x

Neumann[1]

4.0.x[1]

7.6.2

5.2.12

2.3.x

Moore[1]

3.2.x[1]

6.8.12

5.2.12

2.2.x

Lovelace[1]

3.1.x[1]

6.2.2

5.1.19

2.1.x

Kay[1]

3.0.x[1]

5.5.0

5.0.13

2.0.x

Ingalls[1]

2.1.x[1]

2.4.0

4.3.25

1.5.x

Support for upcoming versions of Elasticsearch is being tracked and general compatibility should be given assuming the usage of the high-level REST client.

Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repositories.adoc[] :leveloffset: -1

4. Reference Documentation

4.1. Elasticsearch Clients

This chapter illustrates configuration and usage of supported Elasticsearch client implementations.

Spring Data Elasticsearch operates upon an Elasticsearch client that is connected to a single Elasticsearch node or a cluster. Although the Elasticsearch Client can be used to work with the cluster, applications using Spring Data Elasticsearch normally use the higher level abstractions of Elasticsearch Operations and Elasticsearch Repositories.

4.1.1. High Level REST Client

The Java High Level REST Client is the default client of Elasticsearch, it is configured like shown:

Example 1. High Level REST Client
@Configuration
public class RestClientConfig extends AbstractElasticsearchConfiguration {

    @Override
    @Bean
    public RestHighLevelClient elasticsearchClient() {

        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()  (1)
            .connectedTo("localhost:9200")
            .build();

        return RestClients.create(clientConfiguration).rest();                         (2)
    }
}

// ...

  @Autowired
  RestHighLevelClient highLevelClient;

  RestClient lowLevelClient = highLevelClient.lowLevelClient();                        (3)

// ...

IndexRequest request = new IndexRequest("spring-data")
  .id(randomID())
  .source(singletonMap("feature", "high-level-rest-client"))
  .setRefreshPolicy(IMMEDIATE);

IndexResponse response = highLevelClient.index(request,RequestOptions.DEFAULT);
1 Use the builder to provide cluster addresses, set default HttpHeaders or enable SSL.
2 Create the RestHighLevelClient.
3 It is also possible to obtain the lowLevelRest() client.

4.1.2. Reactive Client

The ReactiveElasticsearchClient is a non official driver based on WebClient. It uses the request/response objects provided by the Elasticsearch core project. Calls are directly operated on the reactive stack, not wrapping async (thread pool bound) responses into reactive types.

Example 2. Reactive REST Client
@Configuration
public class ReactiveRestClientConfig extends AbstractReactiveElasticsearchConfiguration {

    @Override
    @Bean
    public ReactiveElasticsearchClient reactiveElasticsearchClient() {
        final ClientConfiguration clientConfiguration = ClientConfiguration.builder() (1)
            .connectedTo("localhost:9200") //
            .build();
        return ReactiveRestClients.create(clientConfiguration);

    }
}
// ...

Mono<IndexResponse> response = client.index(request ->

  request.index("spring-data")
    .id(randomID())
    .source(singletonMap("feature", "reactive-client"));
);
1 Use the builder to provide cluster addresses, set default HttpHeaders or enable SSL.
The ReactiveClient response, especially for search operations, is bound to the from (offset) & size (limit) options of the request.

4.1.3. Client Configuration

Client behaviour can be changed via the ClientConfiguration that allows to set options for SSL, connect and socket timeouts, headers and other parameters.

Example 3. Client Configuration
HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.add("some-header", "on every request")                      (1)

ClientConfiguration clientConfiguration = ClientConfiguration.builder()
  .connectedTo("localhost:9200", "localhost:9291")                      (2)
  .usingSsl()                                                           (3)
  .withProxy("localhost:8888")                                          (4)
  .withPathPrefix("ela")                                                (5)
  .withConnectTimeout(Duration.ofSeconds(5))                            (6)
  .withSocketTimeout(Duration.ofSeconds(3))                             (7)
  .withDefaultHeaders(defaultHeaders)                                   (8)
  .withBasicAuth(username, password)                                    (9)
  .withHeaders(() -> {                                                  (10)
    HttpHeaders headers = new HttpHeaders();
    headers.add("currentTime", LocalDateTime.now().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
    return headers;
  })
  .withClientConfigurer(                                                (11)
    ReactiveRestClients.WebClientConfigurationCallback.from(webClient -> {
  	  // ...
      return webClient;
  	}))
  .withClientConfigurer(                                                (12)
    RestClients.RestClientConfigurationCallback.from(clientBuilder -> {
  	  // ...
      return clientBuilder;
  	}))
  . // ... other options
  .build();
1 Define default headers, if they need to be customized
2 Use the builder to provide cluster addresses, set default HttpHeaders or enable SSL.
3 Optionally enable SSL.
4 Optionally set a proxy.
5 Optionally set a path prefix, mostly used when different clusters a behind some reverse proxy.
6 Set the connection timeout. Default is 10 sec.
7 Set the socket timeout. Default is 5 sec.
8 Optionally set headers.
9 Add basic authentication.
10 A Supplier<Header> function can be specified which is called every time before a request is sent to Elasticsearch - here, as an example, the current time is written in a header.
11 for reactive setup a function configuring the WebClient
12 for non-reactive setup a function configuring the REST client
Adding a Header supplier as shown in above example allows to inject headers that may change over the time, like authentication JWT tokens. If this is used in the reactive setup, the supplier function must not block!
Elasticsearch 7 compatibility headers

When using Spring Data Elasticsearch 4 - which uses the Elasticsearch 7 client libraries - and accessing an Elasticsearch cluster that is running on version 8, it is necessary to set the compatibility headers see Elasticsearch documentation.

For the imperative client this must be done by setting the default headers, for the reactive code this must be done using a header supplier:

HttpHeaders compatibilityHeaders = new HttpHeaders();
compatibilityHeaders.add("Accept", "application/vnd.elasticsearch+json;compatible-with=7");
compatibilityHeaders.add("Content-Type", "application/vnd.elasticsearch+json;"
    + "compatible-with=7");

ClientConfiguration clientConfiguration = ClientConfiguration.builder()
    .connectedTo("localhost:9200")
    .withProxy("localhost:8080")
    .withBasicAuth("elastic","hcraescitsale")
    .withDefaultHeaders(compatibilityHeaders)    // this variant for imperative code
    .withHeaders(() -> compatibilityHeaders)     // this variant for reactive code
    .build();

4.1.4. Client Logging

To see what is actually sent to and received from the server Request / Response logging on the transport level needs to be turned on as outlined in the snippet below.

Enable transport layer logging
<logger name="org.springframework.data.elasticsearch.client.WIRE" level="trace"/>
The above applies to both the RestHighLevelClient and ReactiveElasticsearchClient when obtained via RestClients respectively ReactiveRestClients.

4.2. Elasticsearch Object Mapping

Spring Data Elasticsearch Object Mapping is the process that maps a Java object - the domain entity - into the JSON representation that is stored in Elasticsearch and back.

Earlier versions of Spring Data Elasticsearch used a Jackson based conversion, Spring Data Elasticsearch 3.2.x introduced the Meta Model Object Mapping. As of version 4.0 only the Meta Object Mapping is used, the Jackson based mapper is not available anymore and the MappingElasticsearchConverter is used.

The main reasons for the removal of the Jackson based mapper are:

  • Custom mappings of fields needed to be done with annotations like @JsonFormat or @JsonInclude. This often caused problems when the same object was used in different JSON based datastores or sent over a JSON based API.

  • Custom field types and formats also need to be stored into the Elasticsearch index mappings. The Jackson based annotations did not fully provide all the information that is necessary to represent the types of Elasticsearch.

  • Fields must be mapped not only when converting from and to entities, but also in query argument, returned data and on other places.

Using the MappingElasticsearchConverter now covers all these cases.

4.2.1. Meta Model Object Mapping

The Metamodel based approach uses domain type information for reading/writing from/to Elasticsearch. This allows to register Converter instances for specific domain type mapping.

Mapping Annotation Overview

The MappingElasticsearchConverter uses metadata to drive the mapping of objects to documents. The metadata is taken from the entity’s properties which can be annotated.

The following annotations are available:

  • @Document: Applied at the class level to indicate this class is a candidate for mapping to the database. The most important attributes are:

    • indexName: the name of the index to store this entity in. This can contain a SpEL template expression like "log-#{T(java.time.LocalDate).now().toString()}"

    • createIndex: flag whether to create an index on repository bootstrapping. Default value is true. See Automatic creation of indices with the corresponding mapping

    • versionType: Configuration of version management. Default value is EXTERNAL.

  • @Id: Applied at the field level to mark the field used for identity purpose.

  • @Transient: By default all fields are mapped to the document when it is stored or retrieved, this annotation excludes the field.

  • @PersistenceConstructor: Marks a given constructor - even a package protected one - to use when instantiating the object from the database. Constructor arguments are mapped by name to the key values in the retrieved Document.

  • @Field: Applied at the field level and defines properties of the field, most of the attributes map to the respective Elasticsearch Mapping definitions (the following list is not complete, check the annotation Javadoc for a complete reference):

    • name: The name of the field as it will be represented in the Elasticsearch document, if not set, the Java field name is used.

    • type: The field type, can be one of Text, Keyword, Long, Integer, Short, Byte, Double, Float, Half_Float, Scaled_Float, Date, Date_Nanos, Boolean, Binary, Integer_Range, Float_Range, Long_Range, Double_Range, Date_Range, Ip_Range, Object, Nested, Ip, TokenCount, Percolator, Flattened, Search_As_You_Type. See Elasticsearch Mapping Types. If the field type is not specified, it defaults to FieldType.Auto. This means, that no mapping entry is written for the property and that Elasticsearch will add a mapping entry dynamically when the first data for this property is stored (check the Elasticsearch documentation for dynamic mapping rules).

    • format: One or more built-in date formats, see the next section Date format mapping.

    • pattern: One or more custom date formats, see the next section Date format mapping.

    • store: Flag whether the original field value should be store in Elasticsearch, default value is false.

    • analyzer, searchAnalyzer, normalizer for specifying custom analyzers and normalizer.

  • @GeoPoint: Marks a field as geo_point datatype. Can be omitted if the field is an instance of the GeoPoint class.

  • @ValueConverter defines a class to be used to convert the given property. In difference to a registered Spring Converter this only converts the annotated property and not every property of the given type.

The mapping metadata infrastructure is defined in a separate spring-data-commons project that is technology agnostic.

Date format mapping

Properties that derive from TemporalAccessor or are of type java.util.Date must either have a @Field annotation of type FieldType.Date or a custom converter must be registered for this type. This paragraph describes the use of FieldType.Date.

There are two attributes of the @Field annotation that define which date format information is written to the mapping (also see Elasticsearch Built In Formats and Elasticsearch Custom Date Formats)

The format attributes is used to define at least one of the predefined formats. If it is not defined, then a default value of _date_optional_time and epoch_millis is used.

The pattern attribute can be used to add additional custom format strings. If you want to use only custom date formats, you must set the format property to empty {}.

The following table shows the different attributes and the mapping created from their values:

annotation format string in Elasticsearch mapping

@Field(type=FieldType.Date)

"date_optional_time||epoch_millis",

@Field(type=FieldType.Date, format=DateFormat.basic_date)

"basic_date"

@Field(type=FieldType.Date, format={DateFormat.basic_date, DateFormat.basic_time})

"basic_date||basic_time"

@Field(type=FieldType.Date, pattern="dd.MM.uuuu")

"date_optional_time||epoch_millis||dd.MM.uuuu",

@Field(type=FieldType.Date, format={}, pattern="dd.MM.uuuu")

"dd.MM.uuuu"

If you are using a custom date format, you need to use uuuu for the year instead of yyyy. This is due to a change in Elasticsearch 7.
Range types

When a field is annotated with a type of one of Integer_Range, Float_Range, Long_Range, Double_Range, Date_Range, or Ip_Range the field must be an instance of a class that will be mapped to an Elasticsearch range, for example:

class SomePersonData {

    @Field(type = FieldType.Integer_Range)
    private ValidAge validAge;

    // getter and setter
}

class ValidAge {
    @Field(name="gte")
    private Integer from;

    @Field(name="lte")
    private Integer to;

    // getter and setter
}

As an alternative Spring Data Elasticsearch provides a Range<T> class so that the previous example can be written as:

class SomePersonData {

    @Field(type = FieldType.Integer_Range)
    private Range<Integer> validAge;

    // getter and setter
}

Supported classes for the type <T> are Integer, Long, Float, Double, Date and classes that implement the TemporalAccessor interface.

Mapped field names

Without further configuration, Spring Data Elasticsearch will use the property name of an object as field name in Elasticsearch. This can be changed for individual field by using the @Field annotation on that property.

It is also possible to define a FieldNamingStrategy in the configuration of the client (Elasticsearch Clients). If for example a SnakeCaseFieldNamingStrategy is configured, the property sampleProperty of the object would be mapped to sample_property in Elasticsearch. A FieldNamingStrategy applies to all entities; it can be overwritten by setting a specific name with @Field on a property.

Mapping Rules
Type Hints

Mapping uses type hints embedded in the document sent to the server to allow generic type mapping. Those type hints are represented as _class attributes within the document and are written for each aggregate root.

Example 4. Type Hints
public class Person {              (1)

  @Id String id;
  String firstname;
  String lastname;
}
{
  "_class" : "com.example.Person", (1)
  "id" : "cb7bef",
  "firstname" : "Sarah",
  "lastname" : "Connor"
}
1 By default the domain types class name is used for the type hint.

Type hints can be configured to hold custom information. Use the @TypeAlias annotation to do so.

Make sure to add types with @TypeAlias to the initial entity set (AbstractElasticsearchConfiguration#getInitialEntitySet) to already have entity information available when first reading data from the store.
Example 5. Type Hints with Alias
@TypeAlias("human")                (1)
public class Person {

  @Id String id;
  // ...
}
{
  "_class" : "human",              (1)
  "id" : ...
}
1 The configured alias is used when writing the entity.
Type hints will not be written for nested Objects unless the properties type is Object, an interface or the actual value type does not match the properties declaration.
Disabling Type Hints

It may be necessary to disable writing of type hints when the index that should be used already exists without having the type hints defined in its mapping and with the mapping mode set to strict. In this case, writing the type hint will produce an error, as the field cannot be added automatically.

Type hints can be disabled for the whole application by overriding the method writeTypeHints() in a configuration class derived from AbstractElasticsearchConfiguration (see Elasticsearch Clients).

As an alternativ they can be disabled for a single index with the @Document annotation:

@Document(indexName = "index", writeTypeHint = WriteTypeHint.FALSE)
We strongly advise against disabling Type Hints. Only do this if you are forced to. Disabling type hints can lead to documents not being retrieved correctly from Elasticsearch in case of polymorphic data or document retrieval may fail completely.
Geospatial Types

Geospatial types like Point & GeoPoint are converted into lat/lon pairs.

Example 6. Geospatial types
public class Address {

  String city, street;
  Point location;
}
{
  "city" : "Los Angeles",
  "street" : "2800 East Observatory Road",
  "location" : { "lat" : 34.118347, "lon" : -118.3026284 }
}
GeoJson Types

Spring Data Elasticsearch supports the GeoJson types by providing an interface GeoJson and implementations for the different geometries. They are mapped to Elasticsearch documents according to the GeoJson specification. The corresponding properties of the entity are specified in the index mappings as geo_shape when the index mappings is written. (check the Elasticsearch documentation as well)

Example 7. GeoJson types
public class Address {

  String city, street;
  GeoJsonPoint location;
}
{
  "city": "Los Angeles",
  "street": "2800 East Observatory Road",
  "location": {
    "type": "Point",
    "coordinates": [-118.3026284, 34.118347]
  }
}

The following GeoJson types are implemented:

  • GeoJsonPoint

  • GeoJsonMultiPoint

  • GeoJsonLineString

  • GeoJsonMultiLineString

  • GeoJsonPolygon

  • GeoJsonMultiPolygon

  • GeoJsonGeometryCollection

Collections

For values inside Collections apply the same mapping rules as for aggregate roots when it comes to type hints and Custom Conversions.

Example 8. Collections
public class Person {

  // ...

  List<Person> friends;

}
{
  // ...

  "friends" : [ { "firstname" : "Kyle", "lastname" : "Reese" } ]
}
Maps

For values inside Maps apply the same mapping rules as for aggregate roots when it comes to type hints and Custom Conversions. However the Map key needs to a String to be processed by Elasticsearch.

Example 9. Collections
public class Person {

  // ...

  Map<String, Address> knownLocations;

}
{
  // ...

  "knownLocations" : {
    "arrivedAt" : {
       "city" : "Los Angeles",
       "street" : "2800 East Observatory Road",
       "location" : { "lat" : 34.118347, "lon" : -118.3026284 }
     }
  }
}
Custom Conversions

Looking at the Configuration from the previous section ElasticsearchCustomConversions allows registering specific rules for mapping domain and simple types.

Example 10. Meta Model Object Mapping Configuration
@Configuration
public class Config extends AbstractElasticsearchConfiguration {

  @Override
  public RestHighLevelClient elasticsearchClient() {
    return RestClients.create(ClientConfiguration.create("localhost:9200")).rest();
  }

  @Bean
  @Override
  public ElasticsearchCustomConversions elasticsearchCustomConversions() {
    return new ElasticsearchCustomConversions(
      Arrays.asList(new AddressToMap(), new MapToAddress()));       (1)
  }

  @WritingConverter                                                 (2)
  static class AddressToMap implements Converter<Address, Map<String, Object>> {

    @Override
    public Map<String, Object> convert(Address source) {

      LinkedHashMap<String, Object> target = new LinkedHashMap<>();
      target.put("ciudad", source.getCity());
      // ...

      return target;
    }
  }

  @ReadingConverter                                                 (3)
  static class MapToAddress implements Converter<Map<String, Object>, Address> {

    @Override
    public Address convert(Map<String, Object> source) {

      // ...
      return address;
    }
  }
}
{
  "ciudad" : "Los Angeles",
  "calle" : "2800 East Observatory Road",
  "localidad" : { "lat" : 34.118347, "lon" : -118.3026284 }
}
1 Add Converter implementations.
2 Set up the Converter used for writing DomainType to Elasticsearch.
3 Set up the Converter used for reading DomainType from search result.

4.3. Elasticsearch Operations

Spring Data Elasticsearch uses several interfaces to define the operations that can be called against an Elasticsearch index (for a description of the reactive interfaces see Reactive Elasticsearch Operations).

  • IndexOperations defines actions on index level like creating or deleting an index.

  • DocumentOperations defines actions to store, update and retrieve entities based on their id.

  • SearchOperations define the actions to search for multiple entities using queries

  • ElasticsearchOperations combines the DocumentOperations and SearchOperations interfaces.

These interfaces correspond to the structuring of the Elasticsearch API.

The default implementations of the interfaces offer:

  • index management functionality.

  • Read/Write mapping support for domain types.

  • A rich query and criteria api.

  • Resource management and Exception translation.

Index management and automatic creation of indices and mappings.

The IndexOperations interface and the provided implementation which can be obtained from an ElasticsearchOperations instance - for example with a call to operations.indexOps(clazz)- give the user the ability to create indices, put mappings or store template and alias information in the Elasticsearch cluster. Details of the index that will be created can be set by using the @Setting annotation, refer to Index settings for further information.

None of these operations are done automatically by the implementations of IndexOperations or ElasticsearchOperations. It is the user’s responsibility to call the methods.

There is support for automatic creation of indices and writing the mappings when using Spring Data Elasticsearch repositories, see Automatic creation of indices with the corresponding mapping

4.3.1. ElasticsearchRestTemplate

The ElasticsearchRestTemplate is an implementation of the ElasticsearchOperations interface using the High Level REST Client.

Example 11. ElasticsearchRestTemplate configuration
@Configuration
public class RestClientConfig extends AbstractElasticsearchConfiguration {
  @Override
  public RestHighLevelClient elasticsearchClient() {       (1)
    return RestClients.create(ClientConfiguration.localhost()).rest();
  }

  // no special bean creation needed                       (2)
}
1 Setting up the High Level REST Client.
2 The base class AbstractElasticsearchConfiguration already provides the elasticsearchTemplate bean.

4.3.2. Usage examples

As both ElasticsearchTemplate and ElasticsearchRestTemplate implement the ElasticsearchOperations interface, the code to use them is not different. The example shows how to use an injected ElasticsearchOperations instance in a Spring REST controller. The decision, if this is using the TransportClient or the RestClient is made by providing the corresponding Bean with one of the configurations shown above.

Example 12. ElasticsearchOperations usage
@RestController
@RequestMapping("/")
public class TestController {

  private  ElasticsearchOperations elasticsearchOperations;

  public TestController(ElasticsearchOperations elasticsearchOperations) { (1)
    this.elasticsearchOperations = elasticsearchOperations;
  }

  @PostMapping("/person")
  public String save(@RequestBody Person person) {                         (2)

    IndexQuery indexQuery = new IndexQueryBuilder()
      .withId(person.getId().toString())
      .withObject(person)
      .build();
    String documentId = elasticsearchOperations.index(indexQuery);
    return documentId;
  }

  @GetMapping("/person/{id}")
  public Person findById(@PathVariable("id")  Long id) {                   (3)
    Person person = elasticsearchOperations
      .queryForObject(GetQuery.getById(id.toString()), Person.class);
    return person;
  }
}
1 Let Spring inject the provided ElasticsearchOperations bean in the constructor.
2 Store some entity in the Elasticsearch cluster.
3 Retrieve the entity with a query by id.

To see the full possibilities of ElasticsearchOperations please refer to the API documentation.

4.3.3. Reactive Elasticsearch Operations

ReactiveElasticsearchOperations is the gateway to executing high level commands against an Elasticsearch cluster using the ReactiveElasticsearchClient.

The ReactiveElasticsearchTemplate is the default implementation of ReactiveElasticsearchOperations.

Reactive Elasticsearch Template

To get started the ReactiveElasticsearchTemplate needs to know about the actual client to work with. Please see Reactive Client for details on the client.

Reactive Template Configuration

The easiest way of setting up the ReactiveElasticsearchTemplate is via AbstractReactiveElasticsearchConfiguration providing dedicated configuration method hooks for base package, the initial entity set etc.

Example 13. The AbstractReactiveElasticsearchConfiguration
@Configuration
public class Config extends AbstractReactiveElasticsearchConfiguration {

  @Bean (1)
  @Override
  public ReactiveElasticsearchClient reactiveElasticsearchClient() {
      // ...
  }
}
1 Configure the client to use. This can be done by ReactiveRestClients or directly via DefaultReactiveElasticsearchClient.
If applicable set default HttpHeaders via the ClientConfiguration of the ReactiveElasticsearchClient. See Client Configuration.
If needed the ReactiveElasticsearchTemplate can be configured with default RefreshPolicy and IndicesOptions that get applied to the related requests by overriding the defaults of refreshPolicy() and indicesOptions().

However one might want to be more in control over the actual components and use a more verbose approach.

Example 14. Configure the ReactiveElasticsearchTemplate
@Configuration
public class Config {

  @Bean (1)
  public ReactiveElasticsearchClient reactiveElasticsearchClient() {
    // ...
  }
  @Bean (2)
  public ElasticsearchConverter elasticsearchConverter() {
    return new MappingElasticsearchConverter(elasticsearchMappingContext());
  }
  @Bean (3)
  public SimpleElasticsearchMappingContext elasticsearchMappingContext() {
    return new SimpleElasticsearchMappingContext();
  }
  @Bean (4)
  public ReactiveElasticsearchOperations reactiveElasticsearchOperations() {
    return new ReactiveElasticsearchTemplate(reactiveElasticsearchClient(), elasticsearchConverter());
  }
}
1 Configure the client to use. This can be done by ReactiveRestClients or directly via DefaultReactiveElasticsearchClient.
2 Set up the ElasticsearchConverter used for domain type mapping utilizing metadata provided by the mapping context.
3 The Elasticsearch specific mapping context for domain type metadata.
4 The actual template based on the client and conversion infrastructure.
Reactive Template Usage

ReactiveElasticsearchTemplate lets you save, find and delete your domain objects and map those objects to documents stored in Elasticsearch.

Consider the following:

Example 15. Use the ReactiveElasticsearchTemplate
@Document(indexName = "marvel")
public class Person {

  private @Id String id;
  private String name;
  private int age;
  // Getter/Setter omitted...
}
template.save(new Person("Bruce Banner", 42))                    (1)
  .doOnNext(System.out::println)
  .flatMap(person -> template.findById(person.id, Person.class)) (2)
  .doOnNext(System.out::println)
  .flatMap(person -> template.delete(person))                    (3)
  .doOnNext(System.out::println)
  .flatMap(id -> template.count(Person.class))                   (4)
  .doOnNext(System.out::println)
  .subscribe(); (5)

The above outputs the following sequence on the console.

> Person(id=QjWCWWcBXiLAnp77ksfR, name=Bruce Banner, age=42)
> Person(id=QjWCWWcBXiLAnp77ksfR, name=Bruce Banner, age=42)
> QjWCWWcBXiLAnp77ksfR
> 0
1 Insert a new Person document into the marvel index under type characters. The id is generated on server side and set into the instance returned.
2 Lookup the Person with matching id in the marvel index under type characters.
3 Delete the Person with matching id, extracted from the given instance, in the marvel index under type characters.
4 Count the total number of documents in the marvel index under type characters.
5 Don’t forget to subscribe().

4.3.4. Search Result Types

When a document is retrieved with the methods of the DocumentOperations interface, just the found entity will be returned. When searching with the methods of the SearchOperations interface, additional information is available for each entity, for example the score or the sortValues of the found entity.

In order to return this information, each entity is wrapped in a SearchHit object that contains this entity-specific additional information. These SearchHit objects themselves are returned within a SearchHits object which additionally contains informations about the whole search like the maxScore or requested aggregations. The following classes and interfaces are now available:

SearchHit<T>

Contains the following information:

  • Id

  • Score

  • Sort Values

  • Highlight fields

  • Inner hits (this is an embedded SearchHits object containing eventually returned inner hits)

  • The retrieved entity of type <T>

SearchHits<T>

Contains the following information:

  • Number of total hits

  • Total hits relation

  • Maximum score

  • A list of SearchHit<T> objects

  • Returned aggregations

  • Returned suggest results

SearchPage<T>

Defines a Spring Data Page that contains a SearchHits<T> element and can be used for paging access using repository methods.

SearchScrollHits<T>

Returned by the low level scroll API functions in ElasticsearchRestTemplate, it enriches a SearchHits<T> with the Elasticsearch scroll id.

SearchHitsIterator<T>

An Iterator returned by the streaming functions of the SearchOperations interface.

ReactiveSearchHits

ReactiveSearchOperations has methods returning a Mono<ReactiveSearchHits<T>>, this contains the same information as a SearchHits<T> object, but will provide the contained SearchHit<T> objects as a Flux<SearchHit<T>> and not as a list.

4.3.5. Queries

Almost all of the methods defined in the SearchOperations and ReactiveSearchOperations interface take a Query parameter that defines the query to execute for searching. Query is an interface and Spring Data Elasticsearch provides three implementations: CriteriaQuery, StringQuery and NativeSearchQuery.

CriteriaQuery

CriteriaQuery based queries allow the creation of queries to search for data without knowing the syntax or basics of Elasticsearch queries. They allow the user to build queries by simply chaining and combining Criteria objects that specifiy the criteria the searched documents must fulfill.

when talking about AND or OR when combining criteria keep in mind, that in Elasticsearch AND are converted to a must condition and OR to a should

Criteria and their usage are best explained by example (let’s assume we have a Book entity with a price property):

Example 16. Get books with a given price
Criteria criteria = new Criteria("price").is(42.0);
Query query = new CriteriaQuery(criteria);

Conditions for the same field can be chained, they will be combined with a logical AND:

Example 17. Get books with a given price
Criteria criteria = new Criteria("price").greaterThan(42.0).lessThan(34.0L);
Query query = new CriteriaQuery(criteria);

When chaining Criteria, by default a AND logic is used:

Example 18. Get all persons with first name James and last name Miller:
Criteria criteria = new Criteria("lastname").is("Miller") (1)
  .and("firstname").is("James")                           (2)
Query query = new CriteriaQuery(criteria);
1 the first Criteria
2 the and() creates a new Criteria and chaines it to the first one.

If you want to create nested queries, you need to use subqueries for this. Let’s assume we want to find all persons with a last name of Miller and a first name of either Jack or John:

Example 19. Nested subqueries
Criteria miller = new Criteria("lastName").is("Miller")  (1)
  .subCriteria(                                          (2)
    new Criteria().or("firstName").is("John")            (3)
      .or("firstName").is("Jack")                        (4)
  );
Query query = new CriteriaQuery(criteria);
1 create a first Criteria for the last name
2 this is combined with AND to a subCriteria
3 This sub Criteria is an OR combination for the first name John
4 and the first name Jack

Please refer to the API documentation of the Criteria class for a complete overview of the different available operations.

StringQuery

This class takes an Elasticsearch query as JSON String. The following code shows a query that searches for persons having the first name "Jack":

Query query = new SearchQuery("{ \"match\": { \"firstname\": { \"query\": \"Jack\" } } } ");
SearchHits<Person> searchHits = operations.search(query, Person.class);

Using StringQuery may be appropriate if you already have an Elasticsearch query to use.

NativeSearchQuery

NativeSearchQuery is the class to use when you have a complex query, or a query that cannot be expressed by using the Criteria API, for example when building queries and using aggregates. It allows to use all the different QueryBuilder implementations from the Elasticsearch library therefore named "native".

The following code shows how to search for persons with a given firstname and for the found documents have a terms aggregation that counts the number of occurences of the lastnames for these persons:

Query query = new NativeSearchQueryBuilder()
    .addAggregation(terms("lastnames").field("lastname").size(10)) //
    .withQuery(QueryBuilders.matchQuery("firstname", firstName))
    .build();

SearchHits<Person> searchHits = operations.search(query, Person.class);

4.4. Elasticsearch Repositories

This chapter includes details of the Elasticsearch repository implementation.

Example 20. The sample Book entity
@Document(indexName="books")
class Book {
    @Id
    private String id;

    @Field(type = FieldType.text)
    private String name;

    @Field(type = FieldType.text)
    private String summary;

    @Field(type = FieldType.Integer)
    private Integer price;

	// getter/setter ...
}

4.4.1. Automatic creation of indices with the corresponding mapping

The @Document annotation has an argument createIndex. If this argument is set to true - which is the default value - Spring Data Elasticsearch will during bootstrapping the repository support on application startup check if the index defined by the @Document annotation exists.

If it does not exist, the index will be created and the mappings derived from the entity’s annotations (see Elasticsearch Object Mapping) will be written to the newly created index. Details of the index that will be created can be set by using the @Setting annotation, refer to Index settings for further information.

4.4.2. Query methods

Query lookup strategies

The Elasticsearch module supports all basic query building feature as string queries, native search queries, criteria based queries or have it being derived from the method name.

Declared queries

Deriving the query from the method name is not always sufficient and/or may result in unreadable method names. In this case one might make use of the @Query annotation (see Using @Query Annotation ).

Query creation

Generally the query creation mechanism for Elasticsearch works as described in [repositories.query-methods]. Here’s a short example of what a Elasticsearch query method translates into:

Example 21. Query creation from method names
interface BookRepository extends Repository<Book, String> {
  List<Book> findByNameAndPrice(String name, Integer price);
}

The method name above will be translated into the following Elasticsearch json query

{
    "query": {
        "bool" : {
            "must" : [
                { "query_string" : { "query" : "?", "fields" : [ "name" ] } },
                { "query_string" : { "query" : "?", "fields" : [ "price" ] } }
            ]
        }
    }
}

A list of supported keywords for Elasticsearch is shown below.

Table 1. Supported keywords inside method names
Keyword Sample Elasticsearch Query String

And

findByNameAndPrice

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } }, { "query_string" : { "query" : "?", "fields" : [ "price" ] } } ] } }}

Or

findByNameOrPrice

{ "query" : { "bool" : { "should" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } }, { "query_string" : { "query" : "?", "fields" : [ "price" ] } } ] } }}

Is

findByName

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } } ] } }}

Not

findByNameNot

{ "query" : { "bool" : { "must_not" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } } ] } }}

Between

findByPriceBetween

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] } }}

LessThan

findByPriceLessThan

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : false } } } ] } }}

LessThanEqual

findByPriceLessThanEqual

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] } }}

GreaterThan

findByPriceGreaterThan

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : false, "include_upper" : true } } } ] } }}

GreaterThanEqual

findByPriceGreaterThan

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : true, "include_upper" : true } } } ] } }}

Before

findByPriceBefore

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] } }}

After

findByPriceAfter

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : true, "include_upper" : true } } } ] } }}

Like

findByNameLike

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }}

StartingWith

findByNameStartingWith

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }}

EndingWith

findByNameEndingWith

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "*?", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }}

Contains/Containing

findByNameContaining

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "*?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }}

In (when annotated as FieldType.Keyword)

findByNameIn(Collection<String>names)

{ "query" : { "bool" : { "must" : [ {"bool" : {"must" : [ {"terms" : {"name" : ["?","?"]}} ] } } ] } }}

In

findByNameIn(Collection<String>names)

{ "query": {"bool": {"must": [{"query_string":{"query": "\"?\" \"?\"", "fields": ["name"]}}]}}}

NotIn (when annotated as FieldType.Keyword)

findByNameNotIn(Collection<String>names)

{ "query" : { "bool" : { "must" : [ {"bool" : {"must_not" : [ {"terms" : {"name" : ["?","?"]}} ] } } ] } }}

NotIn

findByNameNotIn(Collection<String>names)

{"query": {"bool": {"must": [{"query_string": {"query": "NOT(\"?\" \"?\")", "fields": ["name"]}}]}}}

True

findByAvailableTrue

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "true", "fields" : [ "available" ] } } ] } }}

False

findByAvailableFalse

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "false", "fields" : [ "available" ] } } ] } }}

OrderBy

findByAvailableTrueOrderByNameDesc

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "true", "fields" : [ "available" ] } } ] } }, "sort":[{"name":{"order":"desc"}}] }

Exists

findByNameExists

{"query":{"bool":{"must":[{"exists":{"field":"name"}}]}}}

IsNull

findByNameIsNull

{"query":{"bool":{"must_not":[{"exists":{"field":"name"}}]}}}

IsNotNull

findByNameIsNotNull

{"query":{"bool":{"must":[{"exists":{"field":"name"}}]}}}

IsEmpty

findByNameIsEmpty

{"query":{"bool":{"must":[{"bool":{"must":[{"exists":{"field":"name"}}],"must_not":[{"wildcard":{"name":{"wildcard":"*"}}}]}}]}}}

IsNotEmpty

findByNameIsNotEmpty

{"query":{"bool":{"must":[{"wildcard":{"name":{"wildcard":"*"}}}]}}}

Methods names to build Geo-shape queries taking GeoJson parameters are not supported. Use ElasticsearchOperations with CriteriaQuery in a custom repository implementation if you need to have such a function in a repository.
Method return types

Repository methods can be defined to have the following return types for returning multiple Elements:

  • List<T>

  • Stream<T>

  • SearchHits<T>

  • List<SearchHit<T>>

  • Stream<SearchHit<T>>

  • SearchPage<T>

Using @Query Annotation
Example 22. Declare query on the method using the @Query annotation.

The arguments passed to the method can be inserted into placeholders in the query string. the placeholders are of the form ?0, ?1, ?2 etc. for the first, second, third parameter and so on.

interface BookRepository extends ElasticsearchRepository<Book, String> {
    @Query("{\"match\": {\"name\": {\"query\": \"?0\"}}}")
    Page<Book> findByName(String name,Pageable pageable);
}

The String that is set as the annotation argument must be a valid Elasticsearch JSON query. It will be sent to Easticsearch as value of the query element; if for example the function is called with the parameter John, it would produce the following query body:

{
  "query": {
    "match": {
      "name": {
        "query": "John"
      }
    }
  }
}
Example 23. @Query annotation on a method taking a Collection argument

A repository method such as

@Query("{\"ids\": {\"values\": ?0 }}")
List<SampleEntity> getByIds(Collection<String> ids);

would make an IDs query to return all the matching documents. So calling the method with a List of ["id1", "id2", "id3"] would produce the query body

{
  "query": {
    "ids": {
      "values": ["id1", "id2", "id3"]
    }
  }
}

4.4.3. Reactive Elasticsearch Repositories

Reactive Elasticsearch repository support builds on the core repository support explained in [repositories] utilizing operations provided via Reactive Elasticsearch Operations executed by a Reactive Client.

Spring Data Elasticsearch reactive repository support uses Project Reactor as its reactive composition library of choice.

There are 3 main interfaces to be used:

  • ReactiveRepository

  • ReactiveCrudRepository

  • ReactiveSortingRepository

Usage

To access domain objects stored in a Elasticsearch using a Repository, just create an interface for it. Before you can actually go on and do that you will need an entity.

Example 24. Sample Person entity
public class Person {

  @Id
  private String id;
  private String firstname;
  private String lastname;
  private Address address;

  // … getters and setters omitted
}
Please note that the id property needs to be of type String.
Example 25. Basic repository interface to persist Person entities
interface ReactivePersonRepository extends ReactiveSortingRepository<Person, String> {

  Flux<Person> findByFirstname(String firstname);                                   (1)

  Flux<Person> findByFirstname(Publisher<String> firstname);                        (2)

  Flux<Person> findByFirstnameOrderByLastname(String firstname);                    (3)

  Flux<Person> findByFirstname(String firstname, Sort sort);                        (4)

  Flux<Person> findByFirstname(String firstname, Pageable page);                    (5)

  Mono<Person> findByFirstnameAndLastname(String firstname, String lastname);       (6)

  Mono<Person> findFirstByLastname(String lastname);                                (7)

  @Query("{ \"bool\" : { \"must\" : { \"term\" : { \"lastname\" : \"?0\" } } } }")
  Flux<Person> findByLastname(String lastname);                                     (8)

  Mono<Long> countByFirstname(String firstname)                                     (9)

  Mono<Boolean> existsByFirstname(String firstname)                                 (10)

  Mono<Long> deleteByFirstname(String firstname)                                    (11)
}
1 The method shows a query for all people with the given lastname.
2 Finder method awaiting input from Publisher to bind parameter value for firstname.
3 Finder method ordering matching documents by lastname.
4 Finder method ordering matching documents by the expression defined via the Sort parameter.
5 Use Pageable to pass offset and sorting parameters to the database.
6 Finder method concating criteria using And / Or keywords.
7 Find the first matching entity.
8 The method shows a query for all people with the given lastname looked up by running the annotated @Query with given parameters.
9 Count all entities with matching firstname.
10 Check if at least one entity with matching firstname exists.
11 Delete all entites with matching firstname.
Configuration

For Java configuration, use the @EnableReactiveElasticsearchRepositories annotation. If no base package is configured, the infrastructure scans the package of the annotated configuration class.

The following listing shows how to use Java configuration for a repository:

Example 26. Java configuration for repositories
@Configuration
@EnableReactiveElasticsearchRepositories
public class Config extends AbstractReactiveElasticsearchConfiguration {

  @Override
  public ReactiveElasticsearchClient reactiveElasticsearchClient() {
    return ReactiveRestClients.create(ClientConfiguration.localhost());
  }
}

Because the repository from the previous example extends ReactiveSortingRepository, all CRUD operations are available as well as methods for sorted access to the entities. Working with the repository instance is a matter of dependency injecting it into a client, as the following example shows:

Example 27. Sorted access to Person entities
public class PersonRepositoryTests {

  @Autowired ReactivePersonRepository repository;

  @Test
  public void sortsElementsCorrectly() {

    Flux<Person> persons = repository.findAll(Sort.by(new Order(ASC, "lastname")));

    // ...
  }
}

4.4.4. Annotations for repository methods

@Highlight

The @Highlight annotation on a repository method defines for which fields of the returned entity highlighting should be included. To search for some text in a Book 's name or summary and have the found data highlighted, the following repository method can be used:

interface BookRepository extends Repository<Book, String> {

    @Highlight(fields = {
        @HighlightField(name = "name"),
        @HighlightField(name = "summary")
    })
    List<SearchHit<Book>> findByNameOrSummary(String text, String summary);
}

It is possible to define multiple fields to be highlighted like above, and both the @Highlight and the @HighlightField annotation can further be customized with a @HighlightParameters annotation. Check the Javadocs for the possible configuration options.

In the search results the highlight data can be retrieved from the SearchHit class.

4.4.5. Annotation based configuration

The Spring Data Elasticsearch repositories support can be activated using an annotation through JavaConfig.

Example 28. Spring Data Elasticsearch repositories using JavaConfig
@Configuration
@EnableElasticsearchRepositories(                             (1)
  basePackages = "org.springframework.data.elasticsearch.repositories"
  )
static class Config {

  @Bean
  public ElasticsearchOperations elasticsearchTemplate() {    (2)
      // ...
  }
}

class ProductService {

  private ProductRepository repository;                       (3)

  public ProductService(ProductRepository repository) {
    this.repository = repository;
  }

  public Page<Product> findAvailableBookByName(String name, Pageable pageable) {
    return repository.findByAvailableTrueAndNameStartingWith(name, pageable);
  }
}
1 The EnableElasticsearchRepositories annotation activates the Repository support. If no base package is configured, it will use the one of the configuration class it is put on.
2 Provide a Bean named elasticsearchTemplate of type ElasticsearchOperations by using one of the configurations shown in the Elasticsearch Operations chapter.
3 Let Spring inject the Repository bean into your class.

4.4.6. Elasticsearch Repositories using CDI

The Spring Data Elasticsearch repositories can also be set up using CDI functionality.

Example 29. Spring Data Elasticsearch repositories using CDI
class ElasticsearchTemplateProducer {

  @Produces
  @ApplicationScoped
  public ElasticsearchOperations createElasticsearchTemplate() {
    // ...                               (1)
  }
}

class ProductService {

  private ProductRepository repository;  (2)
  public Page<Product> findAvailableBookByName(String name, Pageable pageable) {
    return repository.findByAvailableTrueAndNameStartingWith(name, pageable);
  }
  @Inject
  public void setRepository(ProductRepository repository) {
    this.repository = repository;
  }
}
1 Create a component by using the same calls as are used in the Elasticsearch Operations chapter.
2 Let the CDI framework inject the Repository into your class.

4.4.7. Spring Namespace

The Spring Data Elasticsearch module contains a custom namespace allowing definition of repository beans as well as elements for instantiating a ElasticsearchServer .

Using the repositories element looks up Spring Data repositories as described in [repositories.create-instances] .

Example 30. Setting up Elasticsearch repositories using Namespace
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       https://www.springframework.org/schema/beans/spring-beans-3.1.xsd
       http://www.springframework.org/schema/data/elasticsearch
       https://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch-1.0.xsd">

  <elasticsearch:repositories base-package="com.acme.repositories" />

</beans>

Using the Transport Client or Rest Client element registers an instance of Elasticsearch Server in the context.

Example 31. Transport Client using Namespace
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       https://www.springframework.org/schema/beans/spring-beans-3.1.xsd
       http://www.springframework.org/schema/data/elasticsearch
       https://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch-1.0.xsd">

  <elasticsearch:transport-client id="client" cluster-nodes="localhost:9300,someip:9300" />

</beans>
Example 32. Rest Client using Namespace
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
       xsi:schemaLocation="http://www.springframework.org/schema/data/elasticsearch
       https://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch.xsd
       http://www.springframework.org/schema/beans
       https://www.springframework.org/schema/beans/spring-beans.xsd">

  <elasticsearch:rest-client id="restClient" hosts="http://localhost:9200">

</beans>

Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/auditing.adoc[]

4.4.8. Elasticsearch Auditing

Preparing entities

In order for the auditing code to be able to decide whether an entity instance is new, the entity must implement the Persistable<ID> interface which is defined as follows:

package org.springframework.data.domain;

import org.springframework.lang.Nullable;

public interface Persistable<ID> {
    @Nullable
    ID getId();

    boolean isNew();
}

As the existence of an Id is not a sufficient criterion to determine if an enitity is new in Elasticsearch, additional information is necessary. One way is to use the creation-relevant auditing fields for this decision:

A Person entity might look as follows - omitting getter and setter methods for brevity:

@Document(indexName = "person")
public class Person implements Persistable<Long> {
    @Id private Long id;
    private String lastName;
    private String firstName;
    @CreatedDate
    @Field(type = FieldType.Date, format = DateFormat.basic_date_time)
    private Instant createdDate;
    @CreatedBy
    private String createdBy
    @Field(type = FieldType.Date, format = DateFormat.basic_date_time)
    @LastModifiedDate
    private Instant lastModifiedDate;
    @LastModifiedBy
    private String lastModifiedBy;

    public Long getId() {                                                 (1)
        return id;
    }

    @Override
    public boolean isNew() {
        return id == null || (createdDate == null && createdBy == null);  (2)
    }
}
1 the getter is the required implementation from the interface
2 an object is new if it either has no id or none of fields containing creation attributes are set.
Activating auditing

After the entities have been set up and providing the AuditorAware - or ReactiveAuditorAware - the Auditing must be activated by setting the @EnableElasticsearchAuditing on a configuration class:

@Configuration
@EnableElasticsearchRepositories
@EnableElasticsearchAuditing
class MyConfiguration {
   // configuration code
}

When using the reactive stack this must be:

@Configuration
@EnableReactiveElasticsearchRepositories
@EnableReactiveElasticsearchAuditing
class MyConfiguration {
   // configuration code
}

If your code contains more than one AuditorAware bean for different types, you must provide the name of the bean to use as an argument to the auditorAwareRef parameter of the @EnableElasticsearchAuditing annotation.

Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/entity-callbacks.adoc[] :leveloffset: +1

4.5. Elasticsearch EntityCallbacks

Spring Data Elasticsearch uses the EntityCallback API internally for its auditing support and reacts on the following callbacks:

Table 2. Supported Entity Callbacks
Callback Method Description Order

Reactive/BeforeConvertCallback

onBeforeConvert(T entity, IndexCoordinates index)

Invoked before a domain object is converted to org.springframework.data.elasticsearch.core.document.Document. Can return the entity or a modified entity which then will be converted.

Ordered.LOWEST_PRECEDENCE

Reactive/AfterLoadCallback

onAfterLoad(Document document, Class<T> type, IndexCoordinates indexCoordinates)

Invoked after the result from Elasticsearch has been read into a org.springframework.data.elasticsearch.core.document.Document.

Ordered.LOWEST_PRECEDENCE

Reactive/AfterConvertCallback

onAfterConvert(T entity, Document document, IndexCoordinates indexCoordinates)

Invoked after a domain object is converted from org.springframework.data.elasticsearch.core.document.Document on reading result data from Elasticsearch.

Ordered.LOWEST_PRECEDENCE

Reactive/AuditingEntityCallback

onBeforeConvert(Object entity, IndexCoordinates index)

Marks an auditable entity created or modified

100

Reactive/AfterSaveCallback

T onAfterSave(T entity, IndexCoordinates index)

Invoked after a domain object is saved.

Ordered.LOWEST_PRECEDENCE

4.6. Join-Type implementation

Spring Data Elasticsearch supports the Join data type for creating the corresponding index mappings and for storing the relevant information.

4.6.1. Setting up the data

For an entity to be used in a parent child join relationship, it must have a property of type JoinField which must be annotated. Let’s assume a Statement entity where a statement may be a question, an answer, a comment or a vote (a Builder is also shown in this example, it’s not necessary, but later used in the sample code):

@Document(indexName = "statements")
@Routing("routing")                                                                       (1)
public class Statement {
    @Id
    private String id;

    @Field(type = FieldType.Text)
    private String text;

    @Field(type = FieldType.Keyword)
    private String routing;

    @JoinTypeRelations(
        relations =
            {
                @JoinTypeRelation(parent = "question", children = {"answer", "comment"}), (2)
                @JoinTypeRelation(parent = "answer", children = "vote")                   (3)
            }
    )
    private JoinField<String> relation;                                                   (4)

    private Statement() {
    }

    public static StatementBuilder builder() {
        return new StatementBuilder();
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getRouting() {
        return routing;
    }

    public void setRouting(Routing routing) {
        this.routing = routing;
    }

    public String getText() {
        return text;
    }

    public void setText(String text) {
        this.text = text;
    }

    public JoinField<String> getRelation() {
        return relation;
    }

    public void setRelation(JoinField<String> relation) {
        this.relation = relation;
    }

    public static final class StatementBuilder {
        private String id;
        private String text;
        private String routing;
        private JoinField<String> relation;

        private StatementBuilder() {
        }

        public StatementBuilder withId(String id) {
            this.id = id;
            return this;
        }

        public StatementBuilder withRouting(String routing) {
            this.routing = routing;
            return this;
        }

        public StatementBuilder withText(String text) {
            this.text = text;
            return this;
        }

        public StatementBuilder withRelation(JoinField<String> relation) {
            this.relation = relation;
            return this;
        }

        public Statement build() {
            Statement statement = new Statement();
            statement.setId(id);
            statement.setRouting(routing);
            statement.setText(text);
            statement.setRelation(relation);
            return statement;
        }
    }
}
1 for routing related info see Routing values
2 a question can have answers and comments
3 an answer can have votes
4 the JoinField property is used to combine the name (question, answer, comment or vote) of the relation with the parent id. The generic type must be the same as the @Id annotated property.

Spring Data Elasticsearch will build the following mapping for this class:

{
  "statements": {
    "mappings": {
      "properties": {
        "_class": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "routing": {
          "type": "keyword"
        },
        "relation": {
          "type": "join",
          "eager_global_ordinals": true,
          "relations": {
            "question": [
              "answer",
              "comment"
            ],
            "answer": "vote"
          }
        },
        "text": {
          "type": "text"
        }
      }
    }
  }
}

4.6.2. Storing data

Given a repository for this class the following code inserts a question, two answers, a comment and a vote:

void init() {
    repository.deleteAll();

    Statement savedWeather = repository.save(
        Statement.builder()
            .withText("How is the weather?")
            .withRelation(new JoinField<>("question"))                          (1)
            .build());

    Statement sunnyAnswer = repository.save(
        Statement.builder()
            .withText("sunny")
            .withRelation(new JoinField<>("answer", savedWeather.getId()))      (2)
            .build());

    repository.save(
        Statement.builder()
            .withText("rainy")
            .withRelation(new JoinField<>("answer", savedWeather.getId()))      (3)
            .build());

    repository.save(
        Statement.builder()
            .withText("I don't like the rain")
            .withRelation(new JoinField<>("comment", savedWeather.getId()))     (4)
            .build());

    repository.save(
        Statement.builder()
            .withText("+1 for the sun")
            ,withRouting(savedWeather.getId())
            .withRelation(new JoinField<>("vote", sunnyAnswer.getId()))         (5)
            .build());
}
1 create a question statement
2 the first answer to the question
3 the second answer
4 a comment to the question
5 a vote for the first answer, this needs to have the routing set to the weather document, see Routing values.

4.6.3. Retrieving data

Currently native search queries must be used to query the data, so there is no support from standard repository methods. [repositories.custom-implementations] can be used instead.

The following code shows as an example how to retrieve all entries that have a vote (which must be answers, because only answers can have a vote) using an ElasticsearchOperations instance:

SearchHits<Statement> hasVotes() {
    NativeSearchQuery query = new NativeSearchQueryBuilder()
        .withQuery(hasChildQuery("vote", matchAllQuery(), ScoreMode.None))
        .build();

    return operations.search(query, Statement.class);
}

4.7. Routing values

When Elasticsearch stores a document in an index that has multiple shards, it determines the shard to you use based on the id of the document. Sometimes it is necessary to predefine that multiple documents should be indexed on the same shard (join-types, faster search for related data). For this Elasticsearch offers the possibility to define a routing, which is the value that should be used to calculate the shard from instead of the id.

Spring Data Elasticsearch supports routing definitions on storing and retrieving data in the following ways:

4.7.1. Routing on join-types

When using join-types (see Join-Type implementation), Spring Data Elasticsearch will automatically use the parent property of the entity’s JoinField property as the value for the routing.

This is correct for all the use-cases where the parent-child relationship has just one level. If it is deeper, like a child-parent-grandparent relationship - like in the above example from voteanswerquestion - then the routing needs to explicitly specified by using the techniques described in the next section (the vote needs the question.id as routing value).

4.7.2. Custom routing values

To define a custom routing for an entity, Spring Data Elasticsearch provides a @Routing annotation (reusing the Statement class from above):

@Document(indexName = "statements")
@Routing("routing")                  (1)
public class Statement {
    @Id
    private String id;

    @Field(type = FieldType.Text)
    private String text;

    @JoinTypeRelations(
        relations =
            {
                @JoinTypeRelation(parent = "question", children = {"answer", "comment"}),
                @JoinTypeRelation(parent = "answer", children = "vote")
            }
    )
    private JoinField<String> relation;

    @Nullable
    @Field(type = FieldType.Keyword)
    private String routing;          (2)

    // getter/setter...
}
1 This defines "routing" as routing specification
2 a property with the name routing

If the routing specification of the annotation is a plain string and not a SpEL expression, it is interpreted as the name of a property of the entity, in the example it’s the routing property. The value of this property will then be used as the routing value for all requests that use the entity.

It is also possible to us a SpEL expression in the @Document annotation like this:

@Document(indexName = "statements")
@Routing("@myBean.getRouting(#entity)")
public class Statement{
    // all the needed stuff
}

In this case the user needs to provide a bean with the name myBean that has a method String getRouting(Object). To reference the entity "#entity" must be used in the SpEL expression, and the return value must be null or the routing value as a String.

If plain property’s names and SpEL expressions are not enough to customize the routing definitions, it is possible to define provide an implementation of the RoutingResolver interface. This can then be set on the ElasticOperations instance:

RoutingResolver resolver = ...;

ElasticsearchOperations customOperations= operations.withRouting(resolver);

The withRouting() functions return a copy of the original ElasticsearchOperations instance with the customized routing set.

When a routing has been defined on an entity when it is stored in Elasticsearch, the same value must be provided when doing a get or delete operation. For methods that do not use an entity - like get(ID) or delete(ID) - the ElasticsearchOperations.withRouting(RoutingResolver) method can be used like this:

String id = "someId";
String routing = "theRoutingValue";

// get an entity
Statement s = operations
                .withRouting(RoutingResolver.just(routing))       (1)
                .get(id, Statement.class);

// delete an entity
operations.withRouting(RoutingResolver.just(routing)).delete(id);
1 RoutingResolver.just(s) returns a resolver that will just return the given String.

4.8. Miscellaneous Elasticsearch Operation Support

This chapter covers additional support for Elasticsearch operations that cannot be directly accessed via the repository interface. It is recommended to add those operations as custom implementation as described in [repositories.custom-implementations] .

4.8.1. Index settings

When creating Elasticsearch indices with Spring Data Elasticsearch different index settings can be defined by using the @Setting annotation. The following arguments are available:

  • useServerConfiguration does not send any settings parameters, so the Elasticsearch server configuration determines them.

  • settingPath refers to a JSON file defining the settings that must be resolvable in the classpath

  • shards the number of shards to use, defaults to 1

  • replicas the number of replicas, defaults to 1

  • refreshIntervall, defaults to "1s"

  • indexStoreType, defaults to "fs"

It is as well possible to define index sorting (check the linked Elasticsearch documentation for the possible field types and values):

@Document(indexName = "entities")
@Setting(
  sortFields = { "secondField", "firstField" },                                  (1)
  sortModes = { Setting.SortMode.max, Setting.SortMode.min },                    (2)
  sortOrders = { Setting.SortOrder.desc, Setting.SortOrder.asc },
  sortMissingValues = { Setting.SortMissing._last, Setting.SortMissing._first })
class Entity {
    @Nullable
    @Id private String id;

    @Nullable
    @Field(name = "first_field", type = FieldType.Keyword)
    private String firstField;

    @Nullable @Field(name = "second_field", type = FieldType.Keyword)
    private String secondField;

    // getter and setter...
}
1 when defining sort fields, use the name of the Java property (firstField), not the name that might be defined for Elasticsearch (first_field)
2 sortModes, sortOrders and sortMissingValues are optional, but if they are set, the number of entries must match the number of sortFields elements

4.8.2. Index Mapping

When Spring Data Elasticsearch creates the index mapping with the IndexOperations.createMapping() methods, it uses the annotations described in Mapping Annotation Overview, especially the @Field annotation. In addition to that it is possible to add the @Mapping annotation to a class. This annotation has the following properties:

  • mappingPath a classpath resource in JSON format; if this is not empty it is used as the mapping, no other mapping processing is done.

  • enabled when set to false, this flag is written to the mapping and no further processing is done.

  • dateDetection and numericDetection set the corresponding properties in the mapping when not set to DEFAULT.

  • dynamicDateFormats when this String array is not empty, it defines the date formats used for automatic date detection.

  • runtimeFieldsPath a classpath resource in JSON format containing the definition of runtime fields which is written to the index mappings, for example:

{
  "day_of_week": {
    "type": "keyword",
    "script": {
      "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
    }
  }
}

4.8.3. Filter Builder

Filter Builder improves query speed.

private ElasticsearchOperations operations;

IndexCoordinates index = IndexCoordinates.of("sample-index");

SearchQuery searchQuery = new NativeSearchQueryBuilder()
  .withQuery(matchAllQuery())
  .withFilter(boolFilter().must(termFilter("id", documentId)))
  .build();

Page<SampleEntity> sampleEntities = operations.searchForPage(searchQuery, SampleEntity.class, index);

4.8.4. Using Scroll For Big Result Set

Elasticsearch has a scroll API for getting big result set in chunks. This is internally used by Spring Data Elasticsearch to provide the implementations of the <T> SearchHitsIterator<T> SearchOperations.searchForStream(Query query, Class<T> clazz, IndexCoordinates index) method.

IndexCoordinates index = IndexCoordinates.of("sample-index");

SearchQuery searchQuery = new NativeSearchQueryBuilder()
  .withQuery(matchAllQuery())
  .withFields("message")
  .withPageable(PageRequest.of(0, 10))
  .build();

SearchHitsIterator<SampleEntity> stream = elasticsearchTemplate.searchForStream(searchQuery, SampleEntity.class, index);

List<SampleEntity> sampleEntities = new ArrayList<>();
while (stream.hasNext()) {
  sampleEntities.add(stream.next());
}

stream.close();

There are no methods in the SearchOperations API to access the scroll id, if it should be necessary to access this, the following methods of the ElasticsearchRestTemplate can be used:

@Autowired ElasticsearchRestTemplate template;

IndexCoordinates index = IndexCoordinates.of("sample-index");

SearchQuery searchQuery = new NativeSearchQueryBuilder()
  .withQuery(matchAllQuery())
  .withFields("message")
  .withPageable(PageRequest.of(0, 10))
  .build();

SearchScrollHits<SampleEntity> scroll = template.searchScrollStart(1000, searchQuery, SampleEntity.class, index);

String scrollId = scroll.getScrollId();
List<SampleEntity> sampleEntities = new ArrayList<>();
while (scroll.hasSearchHits()) {
  sampleEntities.addAll(scroll.getSearchHits());
  scrollId = scroll.getScrollId();
  scroll = template.searchScrollContinue(scrollId, 1000, SampleEntity.class);
}
template.searchScrollClear(scrollId);

To use the Scroll API with repository methods, the return type must defined as Stream in the Elasticsearch Repository. The implementation of the method will then use the scroll methods from the ElasticsearchTemplate.

interface SampleEntityRepository extends Repository<SampleEntity, String> {

    Stream<SampleEntity> findBy();

}

4.8.5. Sort options

In addition to the default sort options described in [repositories.paging-and-sorting], Spring Data Elasticsearch provides the class org.springframework.data.elasticsearch.core.query.Order which derives from org.springframework.data.domain.Sort.Order. It offers additional parameters that can be sent to Elasticsearch when specifying the sorting of the result (see https://www.elastic.co/guide/en/elasticsearch/reference/7.15/sort-search-results.html).

There also is the org.springframework.data.elasticsearch.core.query.GeoDistanceOrder class which can be used to have the result of a search operation ordered by geographical distance.

If the class to be retrieved has a GeoPoint property named location, the following Sort would sort the results by distance to the given point:

Sort.by(new GeoDistanceOrder("location", new GeoPoint(48.137154, 11.5761247)))

4.8.6. Runtime Fields

From version 7.12 on Elasticsearch has added the feature of runtime fields (https://www.elastic.co/guide/en/elasticsearch/reference/7.12/runtime.html). Spring Data Elasticsearch supports this in two ways:

Runtime field definitions in the index mappings

The first way to define runtime fields is by adding the definitions to the index mappings (see https://www.elastic.co/guide/en/elasticsearch/reference/7.12/runtime-mapping-fields.html). To use this approach in Spring Data Elasticsearch the user must provide a JSON file that contains the corresponding definition, for example:

Example 33. runtime-fields.json
{
  "day_of_week": {
    "type": "keyword",
    "script": {
      "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
    }
  }
}

The path to this JSON file, which must be present on the classpath, must then be set in the @Mapping annotation of the entity:

@Document(indexName = "runtime-fields")
@Mapping(runtimeFieldsPath = "/runtime-fields.json")
public class RuntimeFieldEntity {
	// properties, getter, setter,...
}
Runtime fields definitions set on a Query

The second way to define runtime fields is by adding the definitions to a search query (see https://www.elastic.co/guide/en/elasticsearch/reference/7.12/runtime-search-request.html). The following code example shows how to do this with Spring Data Elasticsearch :

The entity used is a simple object that has a price property:

@Document(indexName = "some_index_name")
public class SomethingToBuy {

	private @Id @Nullable String id;
	@Nullable @Field(type = FieldType.Text) private String description;
	@Nullable @Field(type = FieldType.Double) private Double price;

	// getter and setter
}

The following query uses a runtime field that calculates a priceWithTax value by adding 19% to the price and uses this value in the search query to find all entities where priceWithTax is higher or equal than a given value:

RuntimeField runtimeField = new RuntimeField("priceWithTax", "double", "emit(doc['price'].value * 1.19)");
Query query = new CriteriaQuery(new Criteria("priceWithTax").greaterThanEqual(16.5));
query.addRuntimeField(runtimeField);

SearchHits<SomethingToBuy> searchHits = operations.search(query, SomethingToBuy.class);

This works with every implementation of the Query interface. :leveloffset: -1

4.9. Appendix

Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repository-namespace-reference.adoc[] Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repository-populator-namespace-reference.adoc[] Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repository-query-keywords-reference.adoc[] Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repository-query-return-types-reference.adoc[]

Appendix E: Migration Guides

Upgrading from 3.2.x to 4.0.x

This section describes breaking changes from version 3.2.x to 4.0.x and how removed features can be replaced by new introduced features.

Removal of the used Jackson Mapper

One of the changes in version 4.0.x is that Spring Data Elasticsearch does not use the Jackson Mapper anymore to map an entity to the JSON representation needed for Elasticsearch (see Elasticsearch Object Mapping). In version 3.2.x the Jackson Mapper was the default that was used. It was possible to switch to the meta-model based converter (named ElasticsearchEntityMapper) by explicitly configuring it (Meta Model Object Mapping).

In version 4.0.x the meta-model based converter is the only one that is available and does not need to be configured explicitly. If you had a custom configuration to enable the meta-model converter by providing a bean like this:

@Bean
@Override
public EntityMapper entityMapper() {

  ElasticsearchEntityMapper entityMapper = new ElasticsearchEntityMapper(
    elasticsearchMappingContext(), new DefaultConversionService()
  );
  entityMapper.setConversions(elasticsearchCustomConversions());

  return entityMapper;
}

You now have to remove this bean, the ElasticsearchEntityMapper interface has been removed.

Entity configuration

Some users had custom Jackson annotations on the entity class, for example in order to define a custom name for the mapped document in Elasticsearch or to configure date conversions. These are not taken into account anymore. The needed functionality is now provided with Spring Data Elasticsearch’s @Field annotation. Please see Mapping Annotation Overview for detailed information.

Removal of implicit index name from query objects

In 3.2.x the different query classes like IndexQuery or SearchQuery had properties that were taking the index name or index names that they were operating upon. If these were not set, the passed in entity was inspected to retrieve the index name that was set in the @Document annotation.
In 4.0.x the index name(s) must now be provided in an additional parameter of type IndexCoordinates. By separating this, it now is possible to use one query object against different indices.

So for example the following code:

IndexQuery indexQuery = new IndexQueryBuilder()
  .withId(person.getId().toString())
  .withObject(person)
  .build();

String documentId = elasticsearchOperations.index(indexQuery);

must be changed to:

IndexCoordinates indexCoordinates = elasticsearchOperations.getIndexCoordinatesFor(person.getClass());

IndexQuery indexQuery = new IndexQueryBuilder()
  .withId(person.getId().toString())
  .withObject(person)
  .build();

String documentId = elasticsearchOperations.index(indexQuery, indexCoordinates);

To make it easier to work with entities and use the index name that is contained in the entitie’s @Document annotation, new methods have been added like DocumentOperations.save(T entity);

The new Operations interfaces

In version 3.2 there was the ElasticsearchOperations interface that defined all the methods for the ElasticsearchTemplate class. In version 4 the functions have been split into different interfaces, aligning these interfaces with the Elasticsearch API:

  • DocumentOperations are the functions related documents like saving, or deleting

  • SearchOperations contains the functions to search in Elasticsearch

  • IndexOperations define the functions to operate on indexes, like index creation or mappings creation.

ElasticsearchOperations now extends DocumentOperations and SearchOperations and has methods get access to an IndexOperations instance.

All the functions from the ElasticsearchOperations interface in version 3.2 that are now moved to the IndexOperations interface are still available, they are marked as deprecated and have default implementations that delegate to the new implementation:
/**
 * Create an index for given indexName.
 *
 * @param indexName the name of the index
 * @return {@literal true} if the index was created
 * @deprecated since 4.0, use {@link IndexOperations#create()}
 */
@Deprecated
default boolean createIndex(String indexName) {
	return indexOps(IndexCoordinates.of(indexName)).create();
}
Deprecations
Methods and classes

Many functions and classes have been deprecated. These functions still work, but the Javadocs show with what they should be replaced.

Example from ElasticsearchOperations
/*
 * Retrieves an object from an index.
 *
 * @param query the query defining the id of the object to get
 * @param clazz the type of the object to be returned
 * @return the found object
 * @deprecated since 4.0, use {@link #get(String, Class, IndexCoordinates)}
 */
@Deprecated
@Nullable
<T> T queryForObject(GetQuery query, Class<T> clazz);
Elasticsearch deprecations

Since version 7 the Elasticsearch TransportClient is deprecated, it will be removed with Elasticsearch version 8. Spring Data Elasticsearch deprecates the ElasticsearchTemplate class which uses the TransportClient in version 4.0.

Mapping types were removed from Elasticsearch 7, they still exist as deprecated values in the Spring Data @Document annotation and the IndexCoordinates class but they are not used anymore internally.

Removals
  • As already described, the ElasticsearchEntityMapper interface has been removed.

  • The SearchQuery interface has been merged into it’s base interface Query, so it’s occurrences can just be replaced with Query.

  • The method org.springframework.data.elasticsearch.core.ElasticsearchOperations.query(SearchQuery query, ResultsExtractor<T> resultsExtractor); and the org.springframework.data.elasticsearch.core.ResultsExtractor interface have been removed. These could be used to parse the result from Elasticsearch for cases in which the response mapping done with the Jackson based mapper was not enough. Since version 4.0, there are the new Search Result Types to return the information from an Elasticsearch response, so there is no need to expose this low level functionality.

  • The low level methods startScroll, continueScroll and clearScroll have been removed from the ElasticsearchOperations interface. For low level scroll API access, there now are searchScrollStart, searchScrollContinue and searchScrollClear methods on the ElasticsearchRestTemplate class.

Upgrading from 4.0.x to 4.1.x

This section describes breaking changes from version 4.0.x to 4.1.x and how removed features can be replaced by new introduced features.

Deprecations
Definition of the id property

It is possible to define a property of en entity as the id property by naming it either id or document. This behaviour is now deprecated and will produce a warning. PLease us the @Id annotation to mark a property as being the id property.

Index mappings

In the ReactiveElasticsearchClient.Indices interface the updateMapping methods are deprecated in favour of the putMapping methods. They do the same, but putMapping is consistent with the naming in the Elasticsearch API:

Alias handling

In the IndexOperations interface the methods addAlias(AliasQuery), removeAlias(AliasQuery) and queryForAlias() have been deprecated. The new methods alias(AliasAction), getAliases(String…​) and getAliasesForIndex(String…​) offer more functionality and a cleaner API.

Parent-ID

Usage of a parent-id has been removed from Elasticsearch since version 6. We now deprecate the corresponding fields and methods.

Removals
Type mappings

The type mappings parameters of the @Document annotation and the IndexCoordinates object were removed. They had been deprecated in Spring Data Elasticsearch 4.0 and their values weren’t used anymore.

Breaking Changes
Return types of ReactiveElasticsearchClient.Indices methods

The methods in the ReactiveElasticsearchClient.Indices were not used up to now. With the introduction of the ReactiveIndexOperations it became necessary to change some of the return types:

  • the createIndex variants now return a Mono<Boolean> instead of a Mono<Void> to signal successful index creation.

  • the updateMapping variants now return a Mono<Boolean> instead of a Mono<Void> to signal successful mappings storage.

Return types of DocumentOperartions.bulkIndex methods

These methods were returing a List<String> containing the ids of the new indexed records. Now they return a List<IndexedObjectInformation>; these objects contain the id and information about optimistic locking (seq_no and primary_term)

Upgrading from 4.1.x to 4.2.x

This section describes breaking changes from version 4.1.x to 4.2.x and how removed features can be replaced by new introduced features.

Deprecations
@Document parameters

The parameters of the @Document annotation that are relevant for the index settings (useServerConfiguration, shards. replicas, refreshIntervall and indexStoretype) have been moved to the @Setting annotation. Use in @Document is still possible but deprecated.

Removals

The @Score annotation that was used to set the score return value in an entity was deprecated in version 4.0 and has been removed. Scroe values are returned in the SearchHit instances that encapsulate the returned entities.

The org.springframework.data.elasticsearch.ElasticsearchException class has been removed. The remaining usages have been replaced with org.springframework.data.mapping.MappingException and org.springframework.dao.InvalidDataAccessApiUsageException.

The deprecated ScoredPage, ScrolledPage @AggregatedPage and implementations has been removed.

The deprecated GetQuery and DeleteQuery have been removed.

The deprecated find methods from ReactiveSearchOperations and ReactiveDocumentOperations have been removed.

Breaking Changes
RefreshPolicy
Enum package changed

It was possible in 4.1 to configure the refresh policy for the ReactiveElasticsearchTemplate by overriding the method AbstractReactiveElasticsearchConfiguration.refreshPolicy() in a custom configuration class. The return value of this method was an instance of the class org.elasticsearch.action.support.WriteRequest.RefreshPolicy.

Now the configuration must return org.springframework.data.elasticsearch.core.RefreshPolicy. This enum has the same values and triggers the same behaviour as before, so only the import statement has to be adjusted.

Refresh behaviour

ElasticsearchOperations and ReactiveElasticsearchOperations now explicitly use the RefreshPolicy set on the template for write requests if not null. If the refresh policy is null, then nothing special is done, so the cluster defaults are used. ElasticsearchOperations was always using the cluster default before this version.

The provided implementations for ElasticsearchRepository and ReactiveElasticsearchRepository will do an explicit refresh when the refresh policy is null. This is the same behaviour as in previous versions. If a refresh policy is set, then it will be used by the repositories as well.

Refresh configuration

When configuring Spring Data Elasticsearch like described in Elasticsearch Clients by using ElasticsearchConfigurationSupport, AbstractElasticsearchConfiguration or AbstractReactiveElasticsearchConfiguration the refresh policy will be initialized to null. Previously the reactive code initialized this to IMMEDIATE, now reactive and non-reactive code show the same behaviour.

Method return types
delete methods that take a Query

The reactive methods previously returned a Mono<Long> with the number of deleted documents, the non reactive versions were void. They now return a Mono<ByQueryResponse> which contains much more detailed information about the deleted documents and errors that might have occurred.

multiget methods

The implementations of multiget previousl only returned the found entities in a List<T> for non-reactive implementations and in a Flux<T> for reactive implementations. If the request contained ids that were not found, the information that these are missing was not available. The user needed to compare the returned ids to the requested ones to find which ones were missing.

Now the multiget methods return a MultiGetItem for every requested id. This contains information about failures (like non existing indices) and the information if the item existed (then it is contained in the `MultiGetItem) or not.

Upgrading from 4.2.x to 4.3.x

This section describes breaking changes from version 4.2.x to 4.3.x and how removed features can be replaced by new introduced features.

Elasticsearch is working on a new Client that will replace the RestHighLevelClient because the RestHighLevelClient uses code from Elasticsearch core libraries which are not Apache 2 licensed anymore. Spring Data Elasticsearch is preparing for this change as well. This means that internally the implementations for the *Operations interfaces need to change - which should be no problem if users program against the interfaces like ElasticsearchOperations or ReactiveElasticsearchOperations. If you are using the implementation classes like ElasticsearchRestTemplate directly, you will need to adapt to these changes.

Spring Data Elasticsearch also removes or replaces the use of classes from the org.elasticsearch packages in it’s API classes and methods, only using them in the implementation where the access to Elasticsearch is implemented. For the user that means, that some enum classes that were used are replaced by enums that live in org.springframework.data.elasticsearch with the same values, these are internally mapped onto the Elasticsearch ones.

Places where classes are used that cannot easily be replaced, this usage is marked as deprecated, we are working on replacements.

Check the sections on Deprecations and Breaking Changes for further details.

Deprecations
suggest methods

In SearchOperations, and so in ElasticsearchOperations as well, the suggest methods taking a org.elasticsearch.search.suggest.SuggestBuilder as argument and returning a org.elasticsearch.action.search.SearchResponse have been deprecated. Use SearchHits<T> search(Query query, Class<T> clazz) instead, passing in a NativeSearchQuery which can contain a SuggestBuilder and read the suggest results from the returned SearchHit<T>.

In ReactiveSearchOperations the new suggest methods return a Mono<org.springframework.data.elasticsearch.core.suggest.response.Suggest> now. Here as well the old methods are deprecated.

Breaking Changes
Removal of org.elasticsearch classes from the API.
  • In the org.springframework.data.elasticsearch.annotations.CompletionContext annotation the property type() has changed from org.elasticsearch.search.suggest.completion.context.ContextMapping.Type to org.springframework.data.elasticsearch.annotations.CompletionContext.ContextMappingType, the available enum values are the same.

  • In the org.springframework.data.elasticsearch.annotations.Document annotation the versionType() property has changed to org.springframework.data.elasticsearch.annotations.Document.VersionType, the available enum values are the same.

  • In the org.springframework.data.elasticsearch.core.query.Query interface the searchType() property has changed to org.springframework.data.elasticsearch.core.query.Query.SearchType, the available enum values are the same.

  • In the org.springframework.data.elasticsearch.core.query.Query interface the return value of timeout() was changed to java.time.Duration.

  • The SearchHits<T>`class does not contain the `org.elasticsearch.search.aggregations.Aggregations anymore. Instead it now contains an instance of the org.springframework.data.elasticsearch.core.AggregationsContainer<T> class where T is the concrete aggregations type from the underlying client that is used. Currently this will be a org .springframework.data.elasticsearch.core.clients.elasticsearch7.ElasticsearchAggregations object; later different implementations will be available. The same change has been done to the ReactiveSearchOperations.aggregate() functions, the now return a Flux<AggregationContainer<?>>. Programs using the aggregations need to be changed to cast the returned value to the appropriate class to further proces it.

  • methods that might have thrown a org.elasticsearch.ElasticsearchStatusException now will throw org.springframework.data.elasticsearch.RestStatusException instead.

Handling of field and sourceFilter properties of Query

Up to version 4.2 the fields property of a Query was interpreted and added to the include list of the sourceFilter. This was not correct, as these are different things for Elasticsearch. This has been corrected. As a consequence code might not work anymore that relies on using fields to specify which fields should be returned from the document’s _source' and should be changed to use the `sourceFilter.

search_type default value

The default value for the search_type in Elasticsearch is query_then_fetch. This now is also set as default value in the Query implementations, it was previously set to dfs_query_then_fetch.

BulkOptions changes

Some properties of the org.springframework.data.elasticsearch.core.query.BulkOptions class have changed their type:

  • the type of the timeout property has been changed to java.time.Duration.

  • the type of the`refreshPolicy` property has been changed to org.springframework.data.elasticsearch.core.RefreshPolicy.

IndicesOptions change

Spring Data Elasticsearch now uses org.springframework.data.elasticsearch.core.query.IndicesOptions instead of org.elasticsearch.action.support.IndicesOptions.

Completion classes

The classes from the package org.springframework.data.elasticsearch.core.completion have been moved to org.springframework.data.elasticsearch.core.suggest.

Other renamings

The org.springframework.data.elasticsearch.core.mapping.ElasticsearchPersistentPropertyConverter interface has been renamed to org.springframework.data.elasticsearch.core.mapping.PropertyValueConverter. Likewise the implementations classes named XXPersistentPropertyConverter have been renamed to XXPropertyValueConverter. :leveloffset: -1 :leveloffset: -1


1. Out of maintenance