org.springframework.data.hadoop.config.annotation.builders

Interface HadoopConfigConfigurer

All Known Implementing Classes:

HadoopConfigBuilder

public interface HadoopConfigConfigurer

HadoopConfigConfigurer is an interface for HadoopConfigBuilder which is exposed to user via SpringHadoopConfigurerAdapter.

Typically configuration is shown below.

 @Configuration
 @EnableHadoop
 static class Config extends SpringHadoopConfigurerAdapter {

   @Override
   public void configure(HadoopConfigConfigurer config) throws Exception {
     config
       .fileSystemUri("hdfs://foo.uri")
       .withResources()
         .resource("classpath:/test-site-1.xml")
         .resource("classpath:/test-site-2.xml")
         .and()
       .withProperties()
         .property("foo", "jee");
   }

 }
 

Author:

Janne Valkealahti

Method Summary

Methods

Modifier and Type	Method and Description
HadoopConfigConfigurer	fileSystemUri(java.lang.String uri) Specify a Hdfs file system uri.
HadoopConfigConfigurer	jobHistoryAddress(java.lang.String address) Specify a MapReduce job history address.
HadoopConfigConfigurer	loadDefaults(boolean loadDefaults) Specify if Hadoop Configuration is initially based on default values.
HadoopConfigConfigurer	resourceManagerAddress(java.lang.String address) Specify a Yarn resource manager address.
PropertiesConfigurer<HadoopConfigConfigurer>	withProperties() Specify configuration options as properties with a PropertiesConfigurer.
ResourceConfigurer<HadoopConfigConfigurer>	withResources() Specify configuration options as resource properties with a ResourceConfigurer.
SecurityConfigurer<HadoopConfigConfigurer>	withSecurity() Specify security options with a SecurityConfigurer.

Method Detail

withResources

ResourceConfigurer<HadoopConfigConfigurer> withResources()
                                                         throws java.lang.Exception

Specify configuration options as resource properties with a ResourceConfigurer.

JavaConfig:

 public void configure(HadoopConfigConfigurer config) throws Exception {
   Properties props = new Properties();
   config
     .withResources()
       .resource("cfg-1.properties")
       .resource("cfg-2.properties")
       .and();
 }
 

XML:

 <hadoop:configuration properties-location="cfg-1.properties, cfg-2.properties"/>
 

Returns:

ResourceConfigurer for chaining

Throws:

java.lang.Exception - if error occurred

withProperties

PropertiesConfigurer<HadoopConfigConfigurer> withProperties()
                                                            throws java.lang.Exception

Specify configuration options as properties with a PropertiesConfigurer.

JavaConfig:

 public void configure(HadoopConfigConfigurer config) throws Exception {
   Properties props = new Properties();
   config
     .withProperties()
       .properties(props)
       .property("myKey1", ",myValue1")
       .and();
 }
 

XML:

 <util:properties id="props" location="props.properties"/>
   <prop key="myKey1">myValue1</prop>
 </util:properties>
 <hadoop:configuration properties-ref="props"/>
 

Returns:

PropertiesConfigurer for chaining

Throws:

java.lang.Exception - if error occurred

withSecurity

SecurityConfigurer<HadoopConfigConfigurer> withSecurity()
                                                        throws java.lang.Exception

Specify security options with a SecurityConfigurer.

JavaConfig:

 public void configure(HadoopConfigConfigurer config) throws Exception {
   config
     .withSecurity()
       .authMethod("kerberos")
       .namenodePrincipal("hdfs/myhost@LOCALDOMAIN")
       .rmManagerPrincipal("yarn/myhost@LOCALDOMAIN");
 }
 

XML:

No equivalent

Returns:

SecurityConfigurer for chaining

Throws:

java.lang.Exception - if error occurred

fileSystemUri

HadoopConfigConfigurer fileSystemUri(java.lang.String uri)

Specify a Hdfs file system uri.

JavaConfig:

 public void configure(HadoopConfigConfigurer config) throws Exception {
   config
     .fileSystemUri("hdfs://myhost:8020");
 }
 

XML:

 <hadoop:configuration file-system-uri="hdfs://myhost:8020"/>
 

Parameters:

uri - The Hdfs uri

Returns:

HadoopConfigConfigurer for chaining

resourceManagerAddress

HadoopConfigConfigurer resourceManagerAddress(java.lang.String address)

Specify a Yarn resource manager address.

JavaConfig:

 public void configure(HadoopConfigConfigurer config) throws Exception {
   config
     .resourceManagerAddress("myRmHost:8032");
 }
 

XML:

 <hadoop:configuration rm-manager-uri="myRmHost:8032"/>
 

Parameters:

address - The Yarn resource manager address

Returns:

HadoopConfigConfigurer for chaining

jobHistoryAddress

HadoopConfigConfigurer jobHistoryAddress(java.lang.String address)

Specify a MapReduce job history address.

JavaConfig:

 public void configure(HadoopConfigConfigurer config) throws Exception {
   config
     .jobHistoryAddress("myJobHistoryHost:10020");
 }
 

XML:

 <hadoop:configuration job-history-uri="myJobHistoryHost:10020"/>
 

Parameters:

address - The Yarn resource manager address

Returns:

HadoopConfigConfigurer for chaining

loadDefaults

HadoopConfigConfigurer loadDefaults(boolean loadDefaults)

Specify if Hadoop Configuration is initially based on default values. Default is true.

JavaConfig:

 public void configure(HadoopConfigConfigurer config) throws Exception {
   config
     .loadDefaults(true);
 }
 

XML:
No equivalent

Parameters:

loadDefaults - The flag if defaults should be loaded

Returns:

HadoopConfigConfigurer for chaining