Spring for Apache Hadoop Reference Manual


Costin Leau


Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Introduction
1. Requirements
2. Additional Resources
II. Spring and Hadoop
3. Hadoop Configuration, MapReduce, and Distributed Cache
3.1. Using the Spring for Apache Hadoop Namespace
3.2. Configuring Hadoop
3.3. Creating a Hadoop Job
3.3.1. Creating a Hadoop Streaming Job
3.4. Running a Hadoop Job
3.4.1. Using the Hadoop Job tasklet
3.5. Running a Hadoop Tool
3.5.1. Replacing Hadoop shell invocations with tool-runner
3.5.2. Using the Hadoop Tool tasklet
3.6. Running a Hadoop Jar
3.6.1. Using the Hadoop Jar tasklet
3.7. Configuring the Hadoop DistributedCache
3.8. Map Reduce Generic Options
4. Working with the Hadoop File System
4.1. Configuring the file-system
4.2. Scripting the Hadoop API
4.2.1. Using scripts
4.3. Scripting implicit variables
4.3.1. Running scripts
4.3.2. Using the Scripting tasklet
4.4. File System Shell (FsShell)
4.4.1. DistCp API
5. Working with HBase
5.1. Data Access Object (DAO) Support
6. Hive integration
6.1. Starting a Hive Server
6.2. Using the Hive Thrift Client
6.3. Using the Hive JDBC Client
6.4. Running a Hive script or query
6.4.1. Using the Hive tasklet
6.5. Interacting with the Hive API
7. Pig support
7.1. Running a Pig script
7.1.1. Using the Pig tasklet
7.2. Interacting with the Pig API
8. Cascading integration
8.1. Using the Cascading tasklet
8.2. Using Scalding
8.3. Spring-specific local Taps
9. Using the runner classes
10. Security Support
10.1. HDFS permissions
10.2. User impersonation (Kerberos)
III. Developing Spring for Apache Hadoop Applications
11. Guidance and Examples
11.1. Scheduling
11.2. Batch Job Listeners
IV. Spring for Apache Hadoop sample applications
12. Sample prerequisites
13. Wordcount sample using the Spring Framework
13.1. Introduction
14. Wordcount sample using Spring Batch
14.1. Introduction
14.2. Basic Spring for Apache Hadoop configuration
14.3. Build and run the sample application
14.4. Run the sample application as a standlone Java application
V. Other Resources
15. Useful Links
VI. Appendices
A. Using Spring for Apache Hadoop with Amazon EMR
A.1. Start up the cluster
A.2. Open an SSH Tunnel as a SOCKS proxy
A.3. Configuring Hadoop to use a SOCKS proxy
A.4. Accessing the file-system
A.5. Shutting down the cluster
A.6. Example configuration
B. Using Spring for Apache Hadoop with EC2/Apache Whirr
B.1. Setting up the Hadoop cluster on EC2 with Apache Whirr
C. Spring for Apache Hadoop Schema