© 2013-2020 The original author(s).

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Preface

The Spring Data Elasticsearch project applies core Spring concepts to the development of solutions using the Elasticsearch Search Engine. It provides:

  • Templates as a high-level abstraction for storing, searching, sorting documents and building aggregations.

  • Repositories which for example enable the user to express queries by defining interfaces having customized method names (for basic information about repositories see [repositories]).

You will notice similarities to the Spring data solr and mongodb support in the Spring Framework.

1. What’s new

1.1. New in Spring Data Elasticsearch 4.1

  • Uses Spring 5.3.

  • Upgrade to Elasticsearch 7.9.3.

  • Improved API for alias management.

  • Introduction of ReactiveIndexOperations for index management.

  • Index templates support.

  • Support for Geo-shape data with GeoJson.

1.2. New in Spring Data Elasticsearch 4.0

  • Uses Spring 5.2.

  • Upgrade to Elasticsearch 7.6.2.

  • Deprecation of TransportClient usage.

  • Implements most of the mapping-types available for the index mappings.

  • Removal of the Jackson ObjectMapper, now using the MappingElasticsearchConverter

  • Cleanup of the API in the *Operations interfaces, grouping and renaming methods so that they match the Elasticsearch API, deprecating the old methods, aligning with other Spring Data modules.

  • Introduction of SearchHit<T> class to represent a found document together with the relevant result metadata for this document (i.e. sortValues).

  • Introduction of the SearchHits<T> class to represent a whole search result together with the metadata for the complete search result (i.e. max_score).

  • Introduction of SearchPage<T> class to represent a paged result containing a SearchHits<T> instance.

  • Introduction of the GeoDistanceOrder class to be able to create sorting by geographical distance

  • Implementation of Auditing Support

  • Implementation of lifecycle entity callbacks

1.3. New in Spring Data Elasticsearch 3.2

2. Project Metadata

3. Requirements

Requires an installation of Elasticsearch.

3.1. Versions

The following table shows the Elasticsearch versions that are used by Spring Data release trains and version of Spring Data Elasticsearch included in that, as well as the Spring Boot versions referring to that particular Spring Data release train:

Spring Data Release Train Spring Data Elasticsearch Elasticsearch Spring Boot

2020.0.0[1]

4.1.x[1]

7.9.3

2.4.x[1]

Neumann

4.0.x

7.6.2

2.3.x

Moore

3.2.x

6.8.12

2.2.x

Lovelace

3.1.x

6.2.2

2.1.x

Kay[2]

3.0.x[2]

5.5.0

2.0.x[2]

Ingalls[2]

2.1.x[2]

2.4.0

1.5.x[2]

Support for upcoming versions of Elasticsearch is being tracked and general compatibility should be given assuming the usage of the high-level REST client.

Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repositories.adoc[] :leveloffset: -1

4. Reference Documentation

4.1. Elasticsearch Clients

This chapter illustrates configuration and usage of supported Elasticsearch client implementations.

Spring Data Elasticsearch operates upon an Elasticsearch client that is connected to a single Elasticsearch node or a cluster. Although the Elasticsearch Client can be used to work with the cluster, applications using Spring Data Elasticsearch normally use the higher level abstractions of Elasticsearch Operations and Elasticsearch Repositories.

4.1.1. Transport Client

The well known TransportClient is deprecated as of Elasticsearch 7 and will be removed in Elasticsearch 8. (see the Elasticsearch documentation). Spring Data Elasticsearch will support the TransportClient as long as it is available in the used Elasticsearch version but has deprecated the classes using it since version 4.0.

We strongly recommend to use the High Level REST Client instead of the TransportClient.

Example 1. Transport Client
@Configuration
public class TransportClientConfig extends ElasticsearchConfigurationSupport {

    @Bean
    public Client elasticsearchClient() throws UnknownHostException {
        Settings settings = Settings.builder().put("cluster.name", "elasticsearch").build();        (1)
        TransportClient client = new PreBuiltTransportClient(settings);
        client.addTransportAddress(new TransportAddress(InetAddress.getByName("127.0.0.1"), 9300)); (2)
        return client;
    }

    @Bean(name = { "elasticsearchOperations", "elasticsearchTemplate" })
    public ElasticsearchTemplate elasticsearchTemplate() throws UnknownHostException {
        return new ElasticsearchTemplate(elasticsearchClient());
    }
}

// ...

IndexRequest request = new IndexRequest("spring-data", "elasticsearch", randomID())
 .source(someObject)
 .setRefreshPolicy(IMMEDIATE);

IndexResponse response = client.index(request);
1 The TransportClient must be configured with the cluster name.
2 The host and port to connect the client to.

4.1.2. High Level REST Client

The Java High Level REST Client is the default client of Elasticsearch, it provides a straight forward replacement for the TransportClient as it accepts and returns the very same request/response objects and therefore depends on the Elasticsearch core project. Asynchronous calls are operated upon a client managed thread pool and require a callback to be notified when the request is done.

Example 2. High Level REST Client
@Configuration
public class RestClientConfig extends AbstractElasticsearchConfiguration {

    @Override
    @Bean
    public RestHighLevelClient elasticsearchClient() {

        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()  (1)
            .connectedTo("localhost:9200")
            .build();

        return RestClients.create(clientConfiguration).rest();                         (2)
    }
}

// ...

  @Autowired
  RestHighLevelClient highLevelClient;

  RestClient lowLevelClient = highLevelClient.lowLevelClient();                        (3)

// ...

IndexRequest request = new IndexRequest("spring-data", "elasticsearch", randomID())
  .source(singletonMap("feature", "high-level-rest-client"))
  .setRefreshPolicy(IMMEDIATE);

IndexResponse response = highLevelClient.index(request);
1 Use the builder to provide cluster addresses, set default HttpHeaders or enable SSL.
2 Create the RestHighLevelClient.
3 It is also possible to obtain the lowLevelRest() client.

4.1.3. Reactive Client

The ReactiveElasticsearchClient is a non official driver based on WebClient. It uses the request/response objects provided by the Elasticsearch core project. Calls are directly operated on the reactive stack, not wrapping async (thread pool bound) responses into reactive types.

Example 3. Reactive REST Client
static class Config {

  @Bean
  ReactiveElasticsearchClient client() {

    ClientConfiguration clientConfiguration = ClientConfiguration.builder()   (1)
      .connectedTo("localhost:9200", "localhost:9291")
      .withWebClientConfigurer(webClient -> {                                 (2)
        ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
            .codecs(configurer -> configurer.defaultCodecs()
                .maxInMemorySize(-1))
            .build();
        return webClient.mutate().exchangeStrategies(exchangeStrategies).build();
       })
      .build();

    return ReactiveRestClients.create(clientConfiguration);
  }
}

// ...

Mono<IndexResponse> response = client.index(request ->

  request.index("spring-data")
    .type("elasticsearch")
    .id(randomID())
    .source(singletonMap("feature", "reactive-client"))
    .setRefreshPolicy(IMMEDIATE);
);
1 Use the builder to provide cluster addresses, set default HttpHeaders or enable SSL.
2 when configuring a reactive client, the withWebClientConfigurer hook can be used to customize the WebClient.
The ReactiveClient response, especially for search operations, is bound to the from (offset) & size (limit) options of the request.

4.1.4. Client Configuration

Client behaviour can be changed via the ClientConfiguration that allows to set options for SSL, connect and socket timeouts, headers and other parameters.

Example 4. Client Configuration
HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.add("some-header", "on every request")                      (1)

ClientConfiguration clientConfiguration = ClientConfiguration.builder()
  .connectedTo("localhost:9200", "localhost:9291")                      (2)
  .usingSsl()                                                           (3)
  .withProxy("localhost:8888")                                          (4)
  .withPathPrefix("ela")                                                (5)
  .withConnectTimeout(Duration.ofSeconds(5))                            (6)
  .withSocketTimeout(Duration.ofSeconds(3))                             (7)
  .withDefaultHeaders(defaultHeaders)                                   (8)
  .withBasicAuth(username, password)                                    (9)
  .withHeaders(() -> {                                                  (10)
    HttpHeaders headers = new HttpHeaders();
    headers.add("currentTime", LocalDateTime.now().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
    return headers;
  })
  . // ... other options
  .build();
1 Define default headers, if they need to be customized
2 Use the builder to provide cluster addresses, set default HttpHeaders or enable SSL.
3 Optionally enable SSL.
4 Optionally set a proxy.
5 Optionally set a path prefix, mostly used when different clusters a behind some reverse proxy.
6 Set the connection timeout. Default is 10 sec.
7 Set the socket timeout. Default is 5 sec.
8 Optionally set headers.
9 Add basic authentication.
10 A Supplier<Header> function can be specified which is called every time before a request is sent to Elasticsearch - here, as an example, the current time is written in a header.
Adding a Header supplier as shown in above example allows to inject headers that may change over the time, like authentication JWT tokens. If this is used in the reactive setup, the supplier function must not block!

4.1.5. Client Logging

To see what is actually sent to and received from the server Request / Response logging on the transport level needs to be turned on as outlined in the snippet below.

Enable transport layer logging
<logger name="org.springframework.data.elasticsearch.client.WIRE" level="trace"/>
The above applies to both the RestHighLevelClient and ReactiveElasticsearchClient when obtained via RestClients respectively ReactiveRestClients, is not available for the TransportClient.

4.2. Elasticsearch Object Mapping

Spring Data Elasticsearch Object Mapping is the process that maps a Java object - the domain entity - into the JSON representation that is stored in Elasticsearch and back.

Earlier versions of Spring Data Elasticsearch used a Jackson based conversion, Spring Data Elasticsearch 3.2.x introduced the Meta Model Object Mapping. As of version 4.0 only the Meta Object Mapping is used, the Jackson based mapper is not available anymore and the MappingElasticsearchConverter is used.

The main reasons for the removal of the Jackson based mapper are:

  • Custom mappings of fields needed to be done with annotations like @JsonFormat or @JsonInclude. This often caused problems when the same object was used in different JSON based datastores or sent over a JSON based API.

  • Custom field types and formats also need to be stored into the Elasticsearch index mappings. The Jackson based annotations did not fully provide all the information that is necessary to represent the types of Elasticsearch.

  • Fields must be mapped not only when converting from and to entities, but also in query argument, returned data and on other places.

Using the MappingElasticsearchConverter now covers all these cases.

4.2.1. Meta Model Object Mapping

The Metamodel based approach uses domain type information for reading/writing from/to Elasticsearch. This allows to register Converter instances for specific domain type mapping.

Mapping Annotation Overview

The MappingElasticsearchConverter uses metadata to drive the mapping of objects to documents. The metadata is taken from the entity’s properties which can be annotated.

The following annotations are available:

  • @Document: Applied at the class level to indicate this class is a candidate for mapping to the database. The most important attributes are:

    • indexName: the name of the index to store this entity in. This can contain a SpEL template expression like "log-#{T(java.time.LocalDate).now().toString()}"

    • type: the mapping type. If not set, the lowercased simple name of the class is used. (deprecated since version 4.0)

    • shards: the number of shards for the index.

    • replicas: the number of replicas for the index.

    • refreshIntervall: Refresh interval for the index. Used for index creation. Default value is "1s".

    • indexStoreType: Index storage type for the index. Used for index creation. Default value is "fs".

    • createIndex: flag whether to create an index on repository bootstrapping. Default value is true. See Automatic creation of indices with the corresponding mapping

    • versionType: Configuration of version management. Default value is EXTERNAL.

  • @Id: Applied at the field level to mark the field used for identity purpose.

  • @Transient: By default all fields are mapped to the document when it is stored or retrieved, this annotation excludes the field.

  • @PersistenceConstructor: Marks a given constructor - even a package protected one - to use when instantiating the object from the database. Constructor arguments are mapped by name to the key values in the retrieved Document.

  • @Field: Applied at the field level and defines properties of the field, most of the attributes map to the respective Elasticsearch Mapping definitions (the following list is not complete, check the annotation Javadoc for a complete reference):

    • name: The name of the field as it will be represented in the Elasticsearch document, if not set, the Java field name is used.

    • type: the field type, can be one of Text, Keyword, Long, Integer, Short, Byte, Double, Float, Half_Float, Scaled_Float, Date, Date_Nanos, Boolean, Binary, Integer_Range, Float_Range, Long_Range, Double_Range, Date_Range, Ip_Range, Object, Nested, Ip, TokenCount, Percolator, Flattened, Search_As_You_Type. See Elasticsearch Mapping Types

    • format and pattern definitions for the Date type.

    • store: Flag whether the original field value should be store in Elasticsearch, default value is false.

    • analyzer, searchAnalyzer, normalizer for specifying custom analyzers and normalizer.

  • @GeoPoint: marks a field as geo_point datatype. Can be omitted if the field is an instance of the GeoPoint class.

Properties that derive from TemporalAccessor or are of type java.util.Date must either have a @Field annotation of type FieldType.Date and a format different from DateFormat.none or a custom converter must be registered for this type.
If you are using a custom date format, you need to use uuuu for the year instead of yyyy. This is due to a change in Elasticsearch 7.

The mapping metadata infrastructure is defined in a separate spring-data-commons project that is technology agnostic.

Mapping Rules
Type Hints

Mapping uses type hints embedded in the document sent to the server to allow generic type mapping. Those type hints are represented as _class attributes within the document and are written for each aggregate root.

Example 5. Type Hints
public class Person {              (1)

  @Id String id;
  String firstname;
  String lastname;
}
{
  "_class" : "com.example.Person", (1)
  "id" : "cb7bef",
  "firstname" : "Sarah",
  "lastname" : "Connor"
}
1 By default the domain types class name is used for the type hint.

Type hints can be configured to hold custom information. Use the @TypeAlias annotation to do so.

Make sure to add types with @TypeAlias to the initial entity set (AbstractElasticsearchConfiguration#getInitialEntitySet) to already have entity information available when first reading data from the store.
Example 6. Type Hints with Alias
@TypeAlias("human")                (1)
public class Person {

  @Id String id;
  // ...
}
{
  "_class" : "human",              (1)
  "id" : ...
}
1 The configured alias is used when writing the entity.
Type hints will not be written for nested Objects unless the properties type is Object, an interface or the actual value type does not match the properties declaration.
Geospatial Types

Geospatial types like Point & GeoPoint are converted into lat/lon pairs.

Example 7. Geospatial types
public class Address {

  String city, street;
  Point location;
}
{
  "city" : "Los Angeles",
  "street" : "2800 East Observatory Road",
  "location" : { "lat" : 34.118347, "lon" : -118.3026284 }
}
GeoJson Types

Spring Data Elasticsearch supports the GeoJson types by providing an interface GeoJson and implementations for the different geometries. They are mapped to Elasticsearch documents according to the GeoJson specification. The corresponding properties of the entity are specified in the index mappings as geo_shape when the index mappings is written. (check the Elasticsearch documentation as well)

Example 8. GeoJson types
public class Address {

  String city, street;
  GeoJsonPoint location;
}
{
  "city": "Los Angeles",
  "street": "2800 East Observatory Road",
  "location": {
    "type": "Point",
    "coordinates": [-118.3026284, 34.118347]
  }
}

The following GeoJson types are implemented:

  • GeoJsonPoint

  • GeoJsonMultiPoint

  • GeoJsonLineString

  • GeoJsonMultiLineString

  • GeoJsonPolygon

  • GeoJsonMultiPolygon

  • GeoJsonGeometryCollection

Collections

For values inside Collections apply the same mapping rules as for aggregate roots when it comes to type hints and Custom Conversions.

Example 9. Collections
public class Person {

  // ...

  List<Person> friends;

}
{
  // ...

  "friends" : [ { "firstname" : "Kyle", "lastname" : "Reese" } ]
}
Maps

For values inside Maps apply the same mapping rules as for aggregate roots when it comes to type hints and Custom Conversions. However the Map key needs to a String to be processed by Elasticsearch.

Example 10. Collections
public class Person {

  // ...

  Map<String, Address> knownLocations;

}
{
  // ...

  "knownLocations" : {
    "arrivedAt" : {
       "city" : "Los Angeles",
       "street" : "2800 East Observatory Road",
       "location" : { "lat" : 34.118347, "lon" : -118.3026284 }
     }
  }
}
Custom Conversions

Looking at the Configuration from the previous section ElasticsearchCustomConversions allows registering specific rules for mapping domain and simple types.

Example 11. Meta Model Object Mapping Configuration
@Configuration
public class Config extends AbstractElasticsearchConfiguration {

  @Override
  public RestHighLevelClient elasticsearchClient() {
    return RestClients.create(ClientConfiguration.create("localhost:9200")).rest();
  }

  @Bean
  @Override
  public ElasticsearchCustomConversions elasticsearchCustomConversions() {
    return new ElasticsearchCustomConversions(
      Arrays.asList(new AddressToMap(), new MapToAddress()));       (1)
  }

  @WritingConverter                                                 (2)
  static class AddressToMap implements Converter<Address, Map<String, Object>> {

    @Override
    public Map<String, Object> convert(Address source) {

      LinkedHashMap<String, Object> target = new LinkedHashMap<>();
      target.put("ciudad", source.getCity());
      // ...

      return target;
    }
  }

  @ReadingConverter                                                 (3)
  static class MapToAddress implements Converter<Map<String, Object>, Address> {

    @Override
    public Address convert(Map<String, Object> source) {

      // ...
      return address;
    }
  }
}
{
  "ciudad" : "Los Angeles",
  "calle" : "2800 East Observatory Road",
  "localidad" : { "lat" : 34.118347, "lon" : -118.3026284 }
}
1 Add Converter implementations.
2 Set up the Converter used for writing DomainType to Elasticsearch.
3 Set up the Converter used for reading DomainType from search result.

4.3. Elasticsearch Operations

Spring Data Elasticsearch uses several interfaces to define the operations that can be called against an Elasticsearch index (for a description of the reactive interfaces see Reactive Elasticsearch Operations).

  • IndexOperations defines actions on index level like creating or deleting an index.

  • DocumentOperations defines actions to store, update and retrieve entities based on their id.

  • SearchOperations define the actions to search for multiple entities using queries

  • ElasticsearchOperations combines the DocumentOperations and SearchOperations interfaces.

These interfaces correspond to the structuring of the Elasticsearch API.

The default implementations of the interfaces offer:

  • index management functionality.

  • Read/Write mapping support for domain types.

  • A rich query and criteria api.

  • Resource management and Exception translation.

Index management and automatic creation of indices and mappings.

The IndexOperations interface and the provided implementation which can be obtained from an ElasticsearchOperations instance - for example with a call to operations.indexOps(clazz)- give the user the ability to create indices, put mappings or store template and alias information in the Elasticsearch cluster.

None of these operations are done automatically by the implementations of IndexOperations or ElasticsearchOperations. It is the user’s responsibility to call the methods.

There is support for automatic creation of indices and writing the mappings when using Spring Data Elasticsearch repositories, see Automatic creation of indices with the corresponding mapping

4.3.1. ElasticsearchTemplate

Usage of the ElasticsearchTemplate is deprecated as of version 4.0, use ElasticsearchRestTemplate instead.

The ElasticsearchTemplate is an implementation of the ElasticsearchOperations interface using the Transport Client.

Example 12. ElasticsearchTemplate configuration
@Configuration
public class TransportClientConfig extends ElasticsearchConfigurationSupport {

  @Bean
  public Client elasticsearchClient() throws UnknownHostException {                 (1)
    Settings settings = Settings.builder().put("cluster.name", "elasticsearch").build();
    TransportClient client = new PreBuiltTransportClient(settings);
    client.addTransportAddress(new TransportAddress(InetAddress.getByName("127.0.0.1"), 9300));
    return client;
  }

  @Bean(name = {"elasticsearchOperations", "elasticsearchTemplate"})
  public ElasticsearchTemplate elasticsearchTemplate() throws UnknownHostException { (2)
  	return new ElasticsearchTemplate(elasticsearchClient());
  }
}
1 Setting up the Transport Client. Deprecated as of version 4.0.
2 Creating the ElasticsearchTemplate bean, offering both names, elasticsearchOperations and elasticsearchTemplate.

4.3.2. ElasticsearchRestTemplate

The ElasticsearchRestTemplate is an implementation of the ElasticsearchOperations interface using the High Level REST Client.

Example 13. ElasticsearchRestTemplate configuration
@Configuration
public class RestClientConfig extends AbstractElasticsearchConfiguration {
  @Override
  public RestHighLevelClient elasticsearchClient() {       (1)
    return RestClients.create(ClientConfiguration.localhost()).rest();
  }

  // no special bean creation needed                       (2)
}
1 Setting up the High Level REST Client.
2 The base class AbstractElasticsearchConfiguration already provides the elasticsearchTemplate bean.

4.3.3. Usage examples

As both ElasticsearchTemplate and ElasticsearchRestTemplate implement the ElasticsearchOperations interface, the code to use them is not different. The example shows how to use an injected ElasticsearchOperations instance in a Spring REST controller. The decision, if this is using the TransportClient or the RestClient is made by providing the corresponding Bean with one of the configurations shown above.

Example 14. ElasticsearchOperations usage
@RestController
@RequestMapping("/")
public class TestController {

  private  ElasticsearchOperations elasticsearchOperations;

  public TestController(ElasticsearchOperations elasticsearchOperations) { (1)
    this.elasticsearchOperations = elasticsearchOperations;
  }

  @PostMapping("/person")
  public String save(@RequestBody Person person) {                         (2)

    IndexQuery indexQuery = new IndexQueryBuilder()
      .withId(person.getId().toString())
      .withObject(person)
      .build();
    String documentId = elasticsearchOperations.index(indexQuery);
    return documentId;
  }

  @GetMapping("/person/{id}")
  public Person findById(@PathVariable("id")  Long id) {                   (3)
    Person person = elasticsearchOperations
      .queryForObject(GetQuery.getById(id.toString()), Person.class);
    return person;
  }
}
1 Let Spring inject the provided ElasticsearchOperations bean in the constructor.
2 Store some entity in the Elasticsearch cluster.
3 Retrieve the entity with a query by id.

To see the full possibilities of ElasticsearchOperations please refer to the API documentation.

4.3.4. Reactive Elasticsearch Operations

ReactiveElasticsearchOperations is the gateway to executing high level commands against an Elasticsearch cluster using the ReactiveElasticsearchClient.

The ReactiveElasticsearchTemplate is the default implementation of ReactiveElasticsearchOperations.

Reactive Elasticsearch Template

To get started the ReactiveElasticsearchTemplate needs to know about the actual client to work with. Please see Reactive Client for details on the client.

Reactive Template Configuration

The easiest way of setting up the ReactiveElasticsearchTemplate is via AbstractReactiveElasticsearchConfiguration providing dedicated configuration method hooks for base package, the initial entity set etc.

Example 15. The AbstractReactiveElasticsearchConfiguration
@Configuration
public class Config extends AbstractReactiveElasticsearchConfiguration {

  @Bean (1)
  @Override
  public ReactiveElasticsearchClient reactiveElasticsearchClient() {
      // ...
  }
}
1 Configure the client to use. This can be done by ReactiveRestClients or directly via DefaultReactiveElasticsearchClient.
If applicable set default HttpHeaders via the ClientConfiguration of the ReactiveElasticsearchClient. See Client Configuration.
If needed the ReactiveElasticsearchTemplate can be configured with default RefreshPolicy and IndicesOptions that get applied to the related requests by overriding the defaults of refreshPolicy() and indicesOptions().

However one might want to be more in control over the actual components and use a more verbose approach.

Example 16. Configure the ReactiveElasticsearchTemplate
@Configuration
public class Config {

  @Bean (1)
  public ReactiveElasticsearchClient reactiveElasticsearchClient() {
    // ...
  }
  @Bean (2)
  public ElasticsearchConverter elasticsearchConverter() {
    return new MappingElasticsearchConverter(elasticsearchMappingContext());
  }
  @Bean (3)
  public SimpleElasticsearchMappingContext elasticsearchMappingContext() {
    return new SimpleElasticsearchMappingContext();
  }
  @Bean (4)
  public ReactiveElasticsearchOperations reactiveElasticsearchOperations() {
    return new ReactiveElasticsearchTemplate(reactiveElasticsearchClient(), elasticsearchConverter());
  }
}
1 Configure the client to use. This can be done by ReactiveRestClients or directly via DefaultReactiveElasticsearchClient.
2 Set up the ElasticsearchConverter used for domain type mapping utilizing metadata provided by the mapping context.
3 The Elasticsearch specific mapping context for domain type metadata.
4 The actual template based on the client and conversion infrastructure.
Reactive Template Usage

ReactiveElasticsearchTemplate lets you save, find and delete your domain objects and map those objects to documents stored in Elasticsearch.

Consider the following:

Example 17. Use the ReactiveElasticsearchTemplate
@Document(indexName = "marvel")
public class Person {

  private @Id String id;
  private String name;
  private int age;
  // Getter/Setter omitted...
}
template.save(new Person("Bruce Banner", 42))                    (1)
  .doOnNext(System.out::println)
  .flatMap(person -> template.findById(person.id, Person.class)) (2)
  .doOnNext(System.out::println)
  .flatMap(person -> template.delete(person))                    (3)
  .doOnNext(System.out::println)
  .flatMap(id -> template.count(Person.class))                   (4)
  .doOnNext(System.out::println)
  .subscribe(); (5)

The above outputs the following sequence on the console.

> Person(id=QjWCWWcBXiLAnp77ksfR, name=Bruce Banner, age=42)
> Person(id=QjWCWWcBXiLAnp77ksfR, name=Bruce Banner, age=42)
> QjWCWWcBXiLAnp77ksfR
> 0
1 Insert a new Person document into the marvel index under type characters. The id is generated on server side and set into the instance returned.
2 Lookup the Person with matching id in the marvel index under type characters.
3 Delete the Person with matching id, extracted from the given instance, in the marvel index under type characters.
4 Count the total number of documents in the marvel index under type characters.
5 Don’t forget to subscribe().

4.3.5. Search Result Types

When a document is retrieved with the methods of the DocumentOperations interface, just the found entity will be returned. When searching with the methods of the SearchOperations interface, additional information is available for each entity, for example the score or the sortValues of the found entity.

In order to return this information, each entity is wrapped in a SearchHit object that contains this entity-specific additional information. These SearchHit objects themselves are returned within a SearchHits object which additionally contains informations about the whole search like the maxScore or requested aggregations. The following classes and interfaces are now available:

SearchHit<T>

Contains the following information:

  • Id

  • Score

  • Sort Values

  • Highlight fields

  • Inner hits (this is an embedded SearchHits object containing eventually returned inner hits)

  • The retrieved entity of type <T>

SearchHits<T>

Contains the following information:

  • Number of total hits

  • Total hits relation

  • Maximum score

  • A list of SearchHit<T> objects

  • Returned aggregations

SearchPage<T>

Defines a Spring Data Page that contains a SearchHits<T> element and can be used for paging access using repository methods.

SearchScrollHits<T>

Returned by the low level scroll API functions in ElasticsearchRestTemplate, it enriches a SearchHits<T> with the Elasticsearch scroll id.

SearchHitsIterator<T>

An Iterator returned by the streaming functions of the SearchOperations interface.

4.3.6. Queries

Almost all of the methods defined in the SearchOperations and ReactiveSearchOperations interface take a Query parameter that defines the query to execute for searching. Query is an interface and Spring Data Elasticsearch provides three implementations: CriteriaQuery, StringQuery and NativeSearchQuery.

CriteriaQuery

CriteriaQuery based queries allow the creation of queries to search for data without knowing the syntax or basics of Elasticsearch queries. They allow the user to build queries by simply chaining and combining Criteria objects that specifiy the criteria the searched documents must fulfill.

when talking about AND or OR when combining criteria keep in mind, that in Elasticsearch AND are converted to a must condition and OR to a should

Criteria and their usage are best explained by example (let’s assume we have a Book entity with a price property):

Example 18. Get books with a given price
Criteria criteria = new Criteria("price").is(42.0);
Query query = new CriteriaQuery(criteria);

Conditions for the same field can be chained, they will be combined with a logical AND:

Example 19. Get books with a given price
Criteria criteria = new Criteria("price").greaterThan(42.0).lessThan(34.0L);
Query query = new CriteriaQuery(criteria);

When chaining Criteria, by default a AND logic is used:

Example 20. Get all persons with first name James and last name Miller:
Criteria criteria = new Criteria("lastname").is("Miller") (1)
  .and("firstname").is("James")                           (2)
Query query = new CriteriaQuery(criteria);
1 the first Criteria
2 the and() creates a new Criteria and chaines it to the first one.

If you want to create nested queries, you need to use subqueries for this. Let’s assume we want to find all persons with a last name of Miller and a first name of either Jack or John:

Example 21. Nested subqueries
Criteria miller = new Criteria("lastName").is("Miller")  (1)
  .subCriteria(                                          (2)
    new Criteria().or("firstName").is("John")            (3)
      .or("firstName").is("Jack")                        (4)
  );
Query query = new CriteriaQuery(criteria);
1 create a first Criteria for the last name
2 this is combined with AND to a subCriteria
3 This sub Criteria is an OR combination for the first name John
4 and the first name Jack

Please refer to the API documentation of the Criteria class for a complete overview of the different available operations.

StringQuery

This class takes an Elasticsearch query as JSON String. The following code shows a query that searches for persons having the first name "Jack":

Query query = new SearchQuery("{ \"match\": { \"firstname\": { \"query\": \"Jack\" } } } ");
SearchHits<Person> searchHits = operations.search(query, Person.class);

Using StringQuery may be appropriate if you already have an Elasticsearch query to use.

NativeSearchQuery

NativeSearchQuery is the class to use when you have a complex query, or a query that cannot be expressed by using the Criteria API, for example when building queries and using aggregates. It allows to use all the different QueryBuilder implementations from the Elasticsearch library therefore named "native".

The following code shows how to search for persons with a given firstname and for the found documents have a terms aggregation that counts the number of occurences of the lastnames for these persons:

Query query = new NativeSearchQueryBuilder()
    .addAggregation(terms("lastnames").field("lastname").size(10)) //
    .withQuery(QueryBuilders.matchQuery("firstname", firstName))
    .build();

SearchHits<Person> searchHits = operations.search(query, Person.class);

4.4. Elasticsearch Repositories

This chapter includes details of the Elasticsearch repository implementation.

Example 22. The sample Book entity
@Document(indexName="books")
class Book {
    @Id
    private String id;

    @Field(type = FieldType.text)
    private String name;

    @Field(type = FieldType.text)
    private String summary;

    @Field(type = FieldType.Integer)
    private Integer price;

	// getter/setter ...
}

4.4.1. Automatic creation of indices with the corresponding mapping

The @Document annotation has an argument createIndex. If this argument is set to true - which is the default value - Spring Data Elasticsearch will during bootstrapping the repository support on application startup check if the index defined by the @Document annotation exists.

If it does not exist, the index will be created and the mappings derived from the entity’s annotations (see Elasticsearch Object Mapping) will be written to the newly created index.

4.4.2. Query methods

Query lookup strategies

The Elasticsearch module supports all basic query building feature as string queries, native search queries, criteria based queries or have it being derived from the method name.

Declared queries

Deriving the query from the method name is not always sufficient and/or may result in unreadable method names. In this case one might make use of the @Query annotation (see Using @Query Annotation ).

Query creation

Generally the query creation mechanism for Elasticsearch works as described in [repositories.query-methods]. Here’s a short example of what a Elasticsearch query method translates into:

Example 23. Query creation from method names
interface BookRepository extends Repository<Book, String> {
  List<Book> findByNameAndPrice(String name, Integer price);
}

The method name above will be translated into the following Elasticsearch json query

{
    "query": {
        "bool" : {
            "must" : [
                { "query_string" : { "query" : "?", "fields" : [ "name" ] } },
                { "query_string" : { "query" : "?", "fields" : [ "price" ] } }
            ]
        }
    }
}

A list of supported keywords for Elasticsearch is shown below.

Table 1. Supported keywords inside method names
Keyword Sample Elasticsearch Query String

And

findByNameAndPrice

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } }, { "query_string" : { "query" : "?", "fields" : [ "price" ] } } ] } }}

Or

findByNameOrPrice

{ "query" : { "bool" : { "should" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } }, { "query_string" : { "query" : "?", "fields" : [ "price" ] } } ] } }}

Is

findByName

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } } ] } }}

Not

findByNameNot

{ "query" : { "bool" : { "must_not" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } } ] } }}

Between

findByPriceBetween

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] } }}

LessThan

findByPriceLessThan

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : false } } } ] } }}

LessThanEqual

findByPriceLessThanEqual

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] } }}

GreaterThan

findByPriceGreaterThan

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : false, "include_upper" : true } } } ] } }}

GreaterThanEqual

findByPriceGreaterThan

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : true, "include_upper" : true } } } ] } }}

Before

findByPriceBefore

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] } }}

After

findByPriceAfter

{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : true, "include_upper" : true } } } ] } }}

Like

findByNameLike

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }}

StartingWith

findByNameStartingWith

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }}

EndingWith

findByNameEndingWith

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "*?", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }}

Contains/Containing

findByNameContaining

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "*?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }}

In (when annotated as FieldType.Keyword)

findByNameIn(Collection<String>names)

{ "query" : { "bool" : { "must" : [ {"bool" : {"must" : [ {"terms" : {"name" : ["?","?"]}} ] } } ] } }}

In

findByNameIn(Collection<String>names)

{ "query": {"bool": {"must": [{"query_string":{"query": "\"?\" \"?\"", "fields": ["name"]}}]}}}

NotIn (when annotated as FieldType.Keyword)

findByNameNotIn(Collection<String>names)

{ "query" : { "bool" : { "must" : [ {"bool" : {"must_not" : [ {"terms" : {"name" : ["?","?"]}} ] } } ] } }}

NotIn

findByNameNotIn(Collection<String>names)

{"query": {"bool": {"must": [{"query_string": {"query": "NOT(\"?\" \"?\")", "fields": ["name"]}}]}}}

Near

findByStoreNear

Not Supported Yet !

True

findByAvailableTrue

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "true", "fields" : [ "available" ] } } ] } }}

False

findByAvailableFalse

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "false", "fields" : [ "available" ] } } ] } }}

OrderBy

findByAvailableTrueOrderByNameDesc

{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "true", "fields" : [ "available" ] } } ] } }, "sort":[{"name":{"order":"desc"}}] }
Methods names to build Geo-shape queries taking GeoJson parameters are not supported. Use ElasticsearchOperations with CriteriaQuery in a custom repository implementation if you need to have such a function in a repository.
Method return types

Repository methods can be defined to have the following return types for returning multiple Elements:

  • List<T>

  • Stream<T>

  • SearchHits<T>

  • List<SearchHit<T>>

  • Stream<SearchHit<T>>

  • SearchPage<T>

Using @Query Annotation
Example 24. Declare query at the method using the @Query annotation.
interface BookRepository extends ElasticsearchRepository<Book, String> {
    @Query("{\"match\": {\"name\": {\"query\": \"?0\"}}}")
    Page<Book> findByName(String name,Pageable pageable);
}

The String that is set as the annotation argument must be a valid Elasticsearch JSON query. It will be sent to Easticsearch as value of the query element; if for example the function is called with the parameter John, it would produce the following query body:

{
  "query": {
    "match": {
      "name": {
        "query": "John"
      }
    }
  }
}

4.4.3. Reactive Elasticsearch Repositories

Reactive Elasticsearch repository support builds on the core repository support explained in [repositories] utilizing operations provided via Reactive Elasticsearch Operations executed by a Reactive Client.

Spring Data Elasticsearch reactive repository support uses Project Reactor as its reactive composition library of choice.

There are 3 main interfaces to be used:

  • ReactiveRepository

  • ReactiveCrudRepository

  • ReactiveSortingRepository

Usage

To access domain objects stored in a Elasticsearch using a Repository, just create an interface for it. Before you can actually go on and do that you will need an entity.

Example 25. Sample Person entity
public class Person {

  @Id
  private String id;
  private String firstname;
  private String lastname;
  private Address address;

  // … getters and setters omitted
}
Please note that the id property needs to be of type String.
Example 26. Basic repository interface to persist Person entities
interface ReactivePersonRepository extends ReactiveSortingRepository<Person, String> {

  Flux<Person> findByFirstname(String firstname);                                   (1)

  Flux<Person> findByFirstname(Publisher<String> firstname);                        (2)

  Flux<Person> findByFirstnameOrderByLastname(String firstname);                    (3)

  Flux<Person> findByFirstname(String firstname, Sort sort);                        (4)

  Flux<Person> findByFirstname(String firstname, Pageable page);                    (5)

  Mono<Person> findByFirstnameAndLastname(String firstname, String lastname);       (6)

  Mono<Person> findFirstByLastname(String lastname);                                (7)

  @Query("{ \"bool\" : { \"must\" : { \"term\" : { \"lastname\" : \"?0\" } } } }")
  Flux<Person> findByLastname(String lastname);                                     (8)

  Mono<Long> countByFirstname(String firstname)                                     (9)

  Mono<Boolean> existsByFirstname(String firstname)                                 (10)

  Mono<Long> deleteByFirstname(String firstname)                                    (11)
}
1 The method shows a query for all people with the given lastname.
2 Finder method awaiting input from Publisher to bind parameter value for firstname.
3 Finder method ordering matching documents by lastname.
4 Finder method ordering matching documents by the expression defined via the Sort parameter.
5 Use Pageable to pass offset and sorting parameters to the database.
6 Finder method concating criteria using And / Or keywords.
7 Find the first matching entity.
8 The method shows a query for all people with the given lastname looked up by running the annotated @Query with given parameters.
9 Count all entities with matching firstname.
10 Check if at least one entity with matching firstname exists.
11 Delete all entites with matching firstname.
Configuration

For Java configuration, use the @EnableReactiveElasticsearchRepositories annotation. If no base package is configured, the infrastructure scans the package of the annotated configuration class.

The following listing shows how to use Java configuration for a repository:

Example 27. Java configuration for repositories
@Configuration
@EnableReactiveElasticsearchRepositories
public class Config extends AbstractReactiveElasticsearchConfiguration {

  @Override
  public ReactiveElasticsearchClient reactiveElasticsearchClient() {
    return ReactiveRestClients.create(ClientConfiguration.localhost());
  }
}

Because the repository from the previous example extends ReactiveSortingRepository, all CRUD operations are available as well as methods for sorted access to the entities. Working with the repository instance is a matter of dependency injecting it into a client, as the following example shows:

Example 28. Sorted access to Person entities
public class PersonRepositoryTests {

  @Autowired ReactivePersonRepository repository;

  @Test
  public void sortsElementsCorrectly() {

    Flux<Person> persons = repository.findAll(Sort.by(new Order(ASC, "lastname")));

    // ...
  }
}

4.4.4. Annotations for repository methods

@Highlight

The @Highlight annotation on a repository method defines for which fields of the returned entity highlighting should be included. To search for some text in a Book 's name or summary and have the found data highlighted, the following repository method can be used:

interface BookRepository extends Repository<Book, String> {

    @Highlight(fields = {
        @HighlightField(name = "name"),
        @HighlightField(name = "summary")
    })
    List<SearchHit<Book>> findByNameOrSummary(String text, String summary);
}

It is possible to define multiple fields to be highlighted like above, and both the @Highlight and the @HighlightField annotation can further be customized with a @HighlightParameters annotation. Check the Javadocs for the possible configuration options.

In the search results the highlight data can be retrieved from the SearchHit class.

4.4.5. Annotation based configuration

The Spring Data Elasticsearch repositories support can be activated using an annotation through JavaConfig.

Example 29. Spring Data Elasticsearch repositories using JavaConfig
@Configuration
@EnableElasticsearchRepositories(                             (1)
  basePackages = "org.springframework.data.elasticsearch.repositories"
  )
static class Config {

  @Bean
  public ElasticsearchOperations elasticsearchTemplate() {    (2)
      // ...
  }
}

class ProductService {

  private ProductRepository repository;                       (3)

  public ProductService(ProductRepository repository) {
    this.repository = repository;
  }

  public Page<Product> findAvailableBookByName(String name, Pageable pageable) {
    return repository.findByAvailableTrueAndNameStartingWith(name, pageable);
  }
}
1 The EnableElasticsearchRepositories annotation activates the Repository support. If no base package is configured, it will use the one of the configuration class it is put on.
2 Provide a Bean named elasticsearchTemplate of type ElasticsearchOperations by using one of the configurations shown in the Elasticsearch Operations chapter.
3 Let Spring inject the Repository bean into your class.

4.4.6. Elasticsearch Repositories using CDI

The Spring Data Elasticsearch repositories can also be set up using CDI functionality.

Example 30. Spring Data Elasticsearch repositories using CDI
class ElasticsearchTemplateProducer {

  @Produces
  @ApplicationScoped
  public ElasticsearchOperations createElasticsearchTemplate() {
    // ...                               (1)
  }
}

class ProductService {

  private ProductRepository repository;  (2)
  public Page<Product> findAvailableBookByName(String name, Pageable pageable) {
    return repository.findByAvailableTrueAndNameStartingWith(name, pageable);
  }
  @Inject
  public void setRepository(ProductRepository repository) {
    this.repository = repository;
  }
}
1 Create a component by using the same calls as are used in the Elasticsearch Operations chapter.
2 Let the CDI framework inject the Repository into your class.

4.4.7. Spring Namespace

The Spring Data Elasticsearch module contains a custom namespace allowing definition of repository beans as well as elements for instantiating a ElasticsearchServer .

Using the repositories element looks up Spring Data repositories as described in [repositories.create-instances] .

Example 31. Setting up Elasticsearch repositories using Namespace
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       https://www.springframework.org/schema/beans/spring-beans-3.1.xsd
       http://www.springframework.org/schema/data/elasticsearch
       https://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch-1.0.xsd">

  <elasticsearch:repositories base-package="com.acme.repositories" />

</beans>

Using the Transport Client or Rest Client element registers an instance of Elasticsearch Server in the context.

Example 32. Transport Client using Namespace
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       https://www.springframework.org/schema/beans/spring-beans-3.1.xsd
       http://www.springframework.org/schema/data/elasticsearch
       https://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch-1.0.xsd">

  <elasticsearch:transport-client id="client" cluster-nodes="localhost:9300,someip:9300" />

</beans>
Example 33. Rest Client using Namespace
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:elasticsearch="http://www.springframework.org/schema/data/elasticsearch"
       xsi:schemaLocation="http://www.springframework.org/schema/data/elasticsearch
       https://www.springframework.org/schema/data/elasticsearch/spring-elasticsearch.xsd
       http://www.springframework.org/schema/beans
       https://www.springframework.org/schema/beans/spring-beans.xsd">

  <elasticsearch:rest-client id="restClient" hosts="http://localhost:9200">

</beans>

Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/auditing.adoc[]

4.4.8. Elasticsearch Auditing

Preparing entities

In order for the auditing code to be able to decide whether an entity instance is new, the entity must implement the Persistable<ID> interface which is defined as follows:

package org.springframework.data.domain;

import org.springframework.lang.Nullable;

public interface Persistable<ID> {
    @Nullable
    ID getId();

    boolean isNew();
}

As the existence of an Id is not a sufficient criterion to determine if an enitity is new in Elasticsearch, additional information is necessary. One way is to use the creation-relevant auditing fields for this decision:

A Person entity might look as follows - omitting getter and setter methods for brevity:

@Document(indexName = "person")
public class Person implements Persistable<Long> {
    @Id private Long id;
    private String lastName;
    private String firstName;
    @CreatedDate
    @Field(type = FieldType.Date, format = DateFormat.basic_date_time)
    private Instant createdDate;
    @CreatedBy
    private String createdBy
    @Field(type = FieldType.Date, format = DateFormat.basic_date_time)
    @LastModifiedDate
    private Instant lastModifiedDate;
    @LastModifiedBy
    private String lastModifiedBy;

    public Long getId() {                                                 (1)
        return id;
    }

    @Override
    public boolean isNew() {
        return id == null || (createdDate == null && createdBy == null);  (2)
    }
}
1 the getter is the required implementation from the interface
2 an object is new if it either has no id or none of fields containing creation attributes are set.
Activating auditing

After the entities have been set up and providing the AuditorAware - or ReactiveAuditorAware - the Auditing must be activated by setting the @EnableElasticsearchAuditing on a configuration class:

@Configuration
@EnableElasticsearchRepositories
@EnableElasticsearchAuditing
class MyConfiguration {
   // configuration code
}

When using the reactive stack this must be:

@Configuration
@EnableReactiveElasticsearchRepositories
@EnableReactiveElasticsearchAuditing
class MyConfiguration {
   // configuration code
}

If your code contains more than one AuditorAware bean for different types, you must provide the name of the bean to use as an argument to the auditorAwareRef parameter of the @EnableElasticsearchAuditing annotation.

Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/entity-callbacks.adoc[] :leveloffset: +1

4.5. Elasticsearch EntityCallbacks

Spring Data Elasticsearch uses the EntityCallback API internally for its auditing support and reacts on the following callbacks:

Table 2. Supported Entity Callbacks
Callback Method Description Order

Reactive/BeforeConvertCallback

onBeforeConvert(T entity, IndexCoordinates index)

Invoked before a domain object is converted to org.springframework.data.elasticsearch.core.document.Document. Can return the entity or a modified entity which then will be converted.

Ordered.LOWEST_PRECEDENCE

Reactive/AfterConvertCallback

onAfterConvert(T entity, Document document, IndexCoordinates indexCoordinates)

Invoked after a domain object is converted from org.springframework.data.elasticsearch.core.document.Document on reading result data from Elasticsearch.

Ordered.LOWEST_PRECEDENCE

Reactive/AuditingEntityCallback

onBeforeConvert(Object entity, IndexCoordinates index)

Marks an auditable entity created or modified

100

Reactive/AfterSaveCallback

T onAfterSave(T entity, IndexCoordinates index)

Invoked after a domain object is saved.

Ordered.LOWEST_PRECEDENCE

4.6. Miscellaneous Elasticsearch Operation Support

This chapter covers additional support for Elasticsearch operations that cannot be directly accessed via the repository interface. It is recommended to add those operations as custom implementation as described in [repositories.custom-implementations] .

4.6.1. Filter Builder

Filter Builder improves query speed.

private ElasticsearchOperations operations;

IndexCoordinates index = IndexCoordinates.of("sample-index");

SearchQuery searchQuery = new NativeSearchQueryBuilder()
  .withQuery(matchAllQuery())
  .withFilter(boolFilter().must(termFilter("id", documentId)))
  .build();

Page<SampleEntity> sampleEntities = operations.searchForPage(searchQuery, SampleEntity.class, index);

4.6.2. Using Scroll For Big Result Set

Elasticsearch has a scroll API for getting big result set in chunks. This is internally used by Spring Data Elasticsearch to provide the implementations of the <T> SearchHitsIterator<T> SearchOperations.searchForStream(Query query, Class<T> clazz, IndexCoordinates index) method.

IndexCoordinates index = IndexCoordinates.of("sample-index");

SearchQuery searchQuery = new NativeSearchQueryBuilder()
  .withQuery(matchAllQuery())
  .withFields("message")
  .withPageable(PageRequest.of(0, 10))
  .build();

SearchHitsIterator<SampleEntity> stream = elasticsearchTemplate.searchForStream(searchQuery, SampleEntity.class, index);

List<SampleEntity> sampleEntities = new ArrayList<>();
while (stream.hasNext()) {
  sampleEntities.add(stream.next());
}

stream.close();

There are no methods in the SearchOperations API to access the scroll id, if it should be necessary to access this, the following methods of the ElasticsearchRestTemplate can be used:

@Autowired ElasticsearchRestTemplate template;

IndexCoordinates index = IndexCoordinates.of("sample-index");

SearchQuery searchQuery = new NativeSearchQueryBuilder()
  .withQuery(matchAllQuery())
  .withFields("message")
  .withPageable(PageRequest.of(0, 10))
  .build();

SearchScrollHits<SampleEntity> scroll = template.searchScrollStart(1000, searchQuery, SampleEntity.class, index);

String scrollId = scroll.getScrollId();
List<SampleEntity> sampleEntities = new ArrayList<>();
while (scroll.hasSearchHits()) {
  sampleEntities.addAll(scroll.getSearchHits());
  scrollId = scroll.getScrollId();
  scroll = template.searchScrollContinue(scrollId, 1000, SampleEntity.class);
}
template.searchScrollClear(scrollId);

To use the Scroll API with repository methods, the return type must defined as Stream in the Elasticsearch Repository. The implementation of the method will then use the scroll methods from the ElasticsearchTemplate.

interface SampleEntityRepository extends Repository<SampleEntity, String> {

    Stream<SampleEntity> findBy();

}

4.6.3. Sort options

In addition to the default sort options described [repositories.paging-and-sorting] Spring Data Elasticsearch has a GeoDistanceOrder class which can be used to have the result of a search operation ordered by geographical distance.

If the class to be retrieved has a GeoPoint property named location, the following Sort would sort the results by distance to the given point:

Sort.by(new GeoDistanceOrder("location", new GeoPoint(48.137154, 11.5761247)))

4.6.4. Join-Type implementation

Spring Data Elasticsearch supports the Join data type for creating the corresponding index mappings and for storing the relevant information.

Setting up the data

For an entity to be used in a parent child join relationship, it must have a property of type JoinField which must be annotated. Let’s assume a Statement entity where a statement may be a question, an answer, a comment or a vote (a Builder is also shown in this example, it’s not necessary, but later used in the sample code):

@Document(indexName = "statements")
public class Statement {
    @Id
    private String id;

    @Field(type = FieldType.Text)
    private String text;

    @JoinTypeRelations(
        relations =
            {
                @JoinTypeRelation(parent = "question", children = {"answer", "comment"}), (1)
                @JoinTypeRelation(parent = "answer", children = "vote")                   (2)
            }
    )
    private JoinField<String> relation;                                                   (3)

    private Statement() {
    }

    public static StatementBuilder builder() {
        return new StatementBuilder();
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getText() {
        return text;
    }

    public void setText(String text) {
        this.text = text;
    }

    public JoinField<String> getRelation() {
        return relation;
    }

    public void setRelation(JoinField<String> relation) {
        this.relation = relation;
    }

    public static final class StatementBuilder {
        private String id;
        private String text;
        private JoinField<String> relation;

        private StatementBuilder() {
        }

        public StatementBuilder withId(String id) {
            this.id = id;
            return this;
        }

        public StatementBuilder withText(String text) {
            this.text = text;
            return this;
        }

        public StatementBuilder withRelation(JoinField<String> relation) {
            this.relation = relation;
            return this;
        }

        public Statement build() {
            Statement statement = new Statement();
            statement.setId(id);
            statement.setText(text);
            statement.setRelation(relation);
            return statement;
        }
    }
}
1 a question can have answers and comments
2 an answer can have votes
3 the JoinField property is used to combine the name (question, answer, comment or vote) of the relation with the parent id. The generic type must be the same as the @Id annotated property.

Spring Data Elasticsearch will build the following mapping for this class:

{
  "statements": {
    "mappings": {
      "properties": {
        "_class": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "relation": {
          "type": "join",
          "eager_global_ordinals": true,
          "relations": {
            "question": [
              "answer",
              "comment"
            ],
            "answer": "vote"
          }
        },
        "text": {
          "type": "text"
        }
      }
    }
  }
}
Storing data

Given a repository for this class the following code inserts a question, two answers, a comment and a vote:

void init() {
    repository.deleteAll();

    Statement savedWeather = repository.save(
        Statement.builder()
            .withText("How is the weather?")
            .withRelation(new JoinField<>("question"))                          (1)
            .build());

    Statement sunnyAnswer = repository.save(
        Statement.builder()
            .withText("sunny")
            .withRelation(new JoinField<>("answer", savedWeather.getId()))      (2)
            .build());

    repository.save(
        Statement.builder()
            .withText("rainy")
            .withRelation(new JoinField<>("answer", savedWeather.getId()))      (3)
            .build());

    repository.save(
        Statement.builder()
            .withText("I don't like the rain")
            .withRelation(new JoinField<>("comment", savedWeather.getId()))     (4)
            .build());

    repository.save(
        Statement.builder()
            .withText("+1 for the sun")
            .withRelation(new JoinField<>("vote", sunnyAnswer.getId()))         (5)
            .build());
}
1 create a question statement
2 the first answer to the question
3 the second answer
4 a comment to the question
5 a vote for the first answer
Retrieving data

Currently native search queries must be used to query the data, so there is no support from standard repository methods. [repositories.custom-implementations] can be used instead.

The following code shows as an example how to retrieve all entries that have a vote (which must be answers, because only answers can have a vote) using an ElasticsearchOperations instance:

SearchHits<Statement> hasVotes() {
    NativeSearchQuery query = new NativeSearchQueryBuilder()
        .withQuery(hasChildQuery("vote", matchAllQuery(), ScoreMode.None))
        .build();

    return operations.search(query, Statement.class);
}

5. Appendix

Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repository-namespace-reference.adoc[] Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repository-populator-namespace-reference.adoc[] Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repository-query-keywords-reference.adoc[] Unresolved directive in index.adoc - include::../../../../spring-data-commons/src/main/asciidoc/repository-query-return-types-reference.adoc[]

Appendix E: Migration Guides

Upgrading from 3.2.x to 4.0.x

This section describes breaking changes from version 3.2.x to 4.0.x and how removed features can be replaced by new introduced features.

Removal of the used Jackson Mapper

One of the changes in version 4.0.x is that Spring Data Elasticsearch does not use the Jackson Mapper anymore to map an entity to the JSON representation needed for Elasticsearch (see Elasticsearch Object Mapping). In version 3.2.x the Jackson Mapper was the default that was used. It was possible to switch to the meta-model based converter (named ElasticsearchEntityMapper) by explicitly configuring it (Meta Model Object Mapping).

In version 4.0.x the meta-model based converter is the only one that is available and does not need to be configured explicitly. If you had a custom configuration to enable the meta-model converter by providing a bean like this:

@Bean
@Override
public EntityMapper entityMapper() {

  ElasticsearchEntityMapper entityMapper = new ElasticsearchEntityMapper(
    elasticsearchMappingContext(), new DefaultConversionService()
  );
  entityMapper.setConversions(elasticsearchCustomConversions());

  return entityMapper;
}

You now have to remove this bean, the ElasticsearchEntityMapper interface has been removed.

Entity configuration

Some users had custom Jackson annotations on the entity class, for example in order to define a custom name for the mapped document in Elasticsearch or to configure date conversions. These are not taken into account anymore. The needed functionality is now provided with Spring Data Elasticsearch’s @Field annotation. Please see Mapping Annotation Overview for detailed information.

Removal of implicit index name from query objects

In 3.2.x the different query classes like IndexQuery or SearchQuery had properties that were taking the index name or index names that they were operating upon. If these were not set, the passed in entity was inspected to retrieve the index name that was set in the @Document annotation.
In 4.0.x the index name(s) must now be provided in an additional parameter of type IndexCoordinates. By separating this, it now is possible to use one query object against different indices.

So for example the following code:

IndexQuery indexQuery = new IndexQueryBuilder()
  .withId(person.getId().toString())
  .withObject(person)
  .build();

String documentId = elasticsearchOperations.index(indexQuery);

must be changed to:

IndexCoordinates indexCoordinates = elasticsearchOperations.getIndexCoordinatesFor(person.getClass());

IndexQuery indexQuery = new IndexQueryBuilder()
  .withId(person.getId().toString())
  .withObject(person)
  .build();

String documentId = elasticsearchOperations.index(indexQuery, indexCoordinates);

To make it easier to work with entities and use the index name that is contained in the entitie’s @Document annotation, new methods have been added like DocumentOperations.save(T entity);

The new Operations interfaces

In version 3.2 there was the ElasticsearchOperations interface that defined all the methods for the ElasticsearchTemplate class. In version 4 the functions have been split into different interfaces, aligning these interfaces with the Elasticsearch API:

  • DocumentOperations are the functions related documents like saving, or deleting

  • SearchOperations contains the functions to search in Elasticsearch

  • IndexOperations define the functions to operate on indexes, like index creation or mappings creation.

ElasticsearchOperations now extends DocumentOperations and SearchOperations and has methods get access to an IndexOperations instance.

All the functions from the ElasticsearchOperations interface in version 3.2 that are now moved to the IndexOperations interface are still available, they are marked as deprecated and have default implementations that delegate to the new implementation: 
/**
 * Create an index for given indexName.
 *
 * @param indexName the name of the index
 * @return {@literal true} if the index was created
 * @deprecated since 4.0, use {@link IndexOperations#create()}
 */
@Deprecated
default boolean createIndex(String indexName) {
	return indexOps(IndexCoordinates.of(indexName)).create();
}
Deprecations
Methods and classes

Many functions and classes have been deprecated. These functions still work, but the Javadocs show with what they should be replaced.

Example from ElasticsearchOperations
/*
 * Retrieves an object from an index.
 *
 * @param query the query defining the id of the object to get
 * @param clazz the type of the object to be returned
 * @return the found object
 * @deprecated since 4.0, use {@link #get(String, Class, IndexCoordinates)}
 */
@Deprecated
@Nullable
<T> T queryForObject(GetQuery query, Class<T> clazz);
Elasticsearch deprecations

Since version 7 the Elasticsearch TransportClient is deprecated, it will be removed with Elasticsearch version 8. Spring Data Elasticsearch deprecates the ElasticsearchTemplate class which uses the TransportClient in version 4.0.

Mapping types were removed from Elasticsearch 7, they still exist as deprecated values in the Spring Data @Document annotation and the IndexCoordinates class but they are not used anymore internally.

Removals

  • As already described, the ElasticsearchEntityMapper interface has been removed.

  • The SearchQuery interface has been merged into it’s base interface Query, so it’s occurrences can just be replaced with Query.

  • The method org.springframework.data.elasticsearch.core.ElasticsearchOperations.query(SearchQuery query, ResultsExtractor<T> resultsExtractor); and the org.springframework.data.elasticsearch.core.ResultsExtractor interface have been removed. These could be used to parse the result from Elasticsearch for cases in which the response mapping done with the Jackson based mapper was not enough. Since version 4.0, there are the new Search Result Types to return the information from an Elasticsearch response, so there is no need to expose this low level functionality.

  • The low level methods startScroll, continueScroll and clearScroll have been removed from the ElasticsearchOperations interface. For low level scroll API access, there now are searchScrollStart, searchScrollContinue and searchScrollClear methods on the ElasticsearchRestTemplate class.

Upgrading from 4.0.x to 4.1.x

This section describes breaking changes from version 4.0.x to 4.1.x and how removed features can be replaced by new introduced features.

Deprecations
Definition of the id property

It is possible to define a property of en entity as the id property by naming it either id or document. This behaviour is now deprecated and will produce a warning. PLease us the @Id annotation to mark a property as being the id property.

Index mappings

In the ReactiveElasticsearchClient.Indices interface the updateMapping methods are deprecated in favour of the putMapping methods. They do the same, but putMapping is consistent with the naming in the Elasticsearch API:

Alias handling

In the IndexOperations interface the methods addAlias(AliasQuery), removeAlias(AliasQuery) and queryForAlias() have been deprecated. The new methods alias(AliasAction), getAliases(String…​) and getAliasesForIndex(String…​) offer more functionality and a cleaner API.

Parent-ID

Usage of a parent-id has been removed from Elasticsearch since version 6. We now deprecate the corresponding fields and methods.

Removals
Type mappings

The type mappings parameters of the @Document annotation and the IndexCoordinates object were removed. They had been deprecated in Spring Data Elasticsearch 4.0 and their values weren’t used anymore.

Breaking Changes
Return types of ReactiveElasticsearchClient.Indices methods

The methods in the ReactiveElasticsearchClient.Indices were not used up to now. With the introduction of the ReactiveIndexOperations it became necessary to change some of the return types:

  • the createIndex variants now return a Mono<Boolean> instead of a Mono<Void> to signal successful index creation.

  • the updateMapping variants now return a Mono<Boolean> instead of a Mono<Void> to signal successful mappings storage.

Return types of DocumentOperartions.bulkIndex methods

These methods were returing a List<String> containing the ids of the new indexed records. Now they return a List<IndexedObjectInformation>; these objects contain the id and information about optimistic locking (seq_no and primary_term) :leveloffset: -1 :leveloffset: -1

1. Currently in development
2. Out of maintenance