23:42:31,620 INFO Test worker context.TestContextManager - @TestExecutionListeners is not present for class [class org.springframework.data.hadoop.mapreduce.JarTests]: using defaults.
23:42:31,623 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:31,733 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:31,758 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:31,868 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:31,868 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@1660d95d: startup date [Wed Sep 25 23:42:31 EDT 2013]; root of context hierarchy
23:42:31,910 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:31,916 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:31,918 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@43afc77: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:31,950 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:31,958 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:32,097 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:32,213 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-71cbffda-49bc-42f9-85d1-e0ca777a6ea6:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:32,213 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-71cbffda-49bc-42f9-85d1-e0ca777a6ea6:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1034368743
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-71cbffda-49bc-42f9-85d1-e0ca777a6ea6
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:32,243 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:32,243 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:32,246 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:32,250 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:32,251 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:32,339 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-69226593-65bb-43dd-8548-c66c19cdcadc:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:32,339 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-69226593-65bb-43dd-8548-c66c19cdcadc:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@842846238
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-69226593-65bb-43dd-8548-c66c19cdcadc
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:32,347 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:32,347 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:32,349 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:32,353 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:32,353 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:32,399 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-8f96c8da-db57-469b-924f-fd6da0513817:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:32,399 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-8f96c8da-db57-469b-924f-fd6da0513817:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:32,418 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:32,419 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:32,420 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:32,422 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:32,442 INFO Test worker support.SimpleJobLauncher - Job: [FlowJob: [name=mainJob]] launched with the following parameters: [{}]
23:42:32,449 INFO Test worker job.SimpleStepHandler - Executing step: [cleanup]
23:42:32,475 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:32,478 INFO Test worker mapreduce.JarTasklet - Invoking [class org.apache.hadoop.examples.ExampleDriver] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/mini-hadoop-examples.jar] with args [[wordcount, /ide-test/input/word, /ide-test/output/wc]]
23:42:32,615 INFO Test worker job.SimpleStepHandler - Executing step: [wordcount]
23:42:32,633 INFO Test worker support.SimpleJobLauncher - Job: [FlowJob: [name=mainJob]] completed with the following parameters: [{}] and the following status: [COMPLETED]
23:42:32,633 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@1660d95d: startup date [Wed Sep 25 23:42:31 EDT 2013]; root of context hierarchy
23:42:32,634 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@43afc77: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:32,638 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:32,665 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:32,684 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:32,694 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:32,695 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@5124e852: startup date [Wed Sep 25 23:42:32 EDT 2013]; root of context hierarchy
23:42:32,704 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:32,710 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:32,712 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1bbe141a: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:32,733 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:32,733 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:32,740 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:32,868 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:32,912 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-b409dced-53ee-49b5-ad8a-a2c01df44ab1:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:32,912 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-b409dced-53ee-49b5-ad8a-a2c01df44ab1:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@298487900
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-b409dced-53ee-49b5-ad8a-a2c01df44ab1
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:32,920 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:32,920 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:32,924 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:32,927 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:32,927 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:32,973 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-9f1402cc-36a8-4092-a06d-f3f90723a4d2:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:32,973 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-9f1402cc-36a8-4092-a06d-f3f90723a4d2:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1260350552
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-9f1402cc-36a8-4092-a06d-f3f90723a4d2
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:32,980 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:32,980 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:32,984 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:32,989 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:32,990 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,027 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-4e92f0df-be8b-43ee-8112-4a2bc3a97ae6:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,027 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-4e92f0df-be8b-43ee-8112-4a2bc3a97ae6:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:33,036 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,036 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,037 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,039 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,053 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@5124e852: startup date [Wed Sep 25 23:42:32 EDT 2013]; root of context hierarchy
23:42:33,053 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1bbe141a: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:33,057 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:33,085 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:33,109 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:33,119 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:33,120 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@55432a38: startup date [Wed Sep 25 23:42:33 EDT 2013]; root of context hierarchy
23:42:33,131 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:33,137 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:33,140 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7b665677: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:33,163 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:33,163 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,170 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,186 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:33,189 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:33,259 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-3b415d46-ffa6-4c9b-a424-f5aff0c94a32:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,259 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-3b415d46-ffa6-4c9b-a424-f5aff0c94a32:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1002507149
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-3b415d46-ffa6-4c9b-a424-f5aff0c94a32
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:33,267 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:33,267 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,269 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:33,272 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:33,273 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,306 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-6e47fc98-7b7d-4ab4-80d9-da26339d06f0:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,306 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-6e47fc98-7b7d-4ab4-80d9-da26339d06f0:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1667514284
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-6e47fc98-7b7d-4ab4-80d9-da26339d06f0
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:33,314 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:33,314 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,316 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:33,319 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:33,320 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,354 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-87fc245c-5a82-4c4d-8301-451002b06c91:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,354 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-87fc245c-5a82-4c4d-8301-451002b06c91:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:33,364 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,364 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,365 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,366 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,380 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@55432a38: startup date [Wed Sep 25 23:42:33 EDT 2013]; root of context hierarchy
23:42:33,381 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7b665677: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:33,386 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:33,412 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:33,432 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:33,443 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:33,443 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@22425573: startup date [Wed Sep 25 23:42:33 EDT 2013]; root of context hierarchy
23:42:33,453 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:33,459 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:33,461 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@a38bd55: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:33,481 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:33,481 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,487 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,504 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:33,507 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:33,546 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-7878d740-db9e-4e3b-9c16-2181f1cea629:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,546 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-7878d740-db9e-4e3b-9c16-2181f1cea629:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@287276761
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-7878d740-db9e-4e3b-9c16-2181f1cea629
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:33,554 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:33,554 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,558 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:33,562 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:33,562 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,608 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-778af343-b794-40be-b30a-5a6e753bf41f:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,609 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-778af343-b794-40be-b30a-5a6e753bf41f:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1057261026
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-778af343-b794-40be-b30a-5a6e753bf41f
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:33,617 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:33,617 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,620 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:33,623 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:33,624 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,659 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-8705a634-4ce4-45bd-b8a0-edd635b8b43a:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,659 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-8705a634-4ce4-45bd-b8a0-edd635b8b43a:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:33,668 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,668 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,670 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,672 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,707 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@22425573: startup date [Wed Sep 25 23:42:33 EDT 2013]; root of context hierarchy
23:42:33,708 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@a38bd55: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:33,711 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:33,734 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:33,753 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:33,763 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:33,763 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@40a775f4: startup date [Wed Sep 25 23:42:33 EDT 2013]; root of context hierarchy
23:42:33,772 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:33,777 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:33,779 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@47487fd9: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:33,796 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:33,797 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,803 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,824 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:33,827 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:33,861 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-5d8c8a7f-da55-4f59-bb0c-6b2520237c06:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,861 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-5d8c8a7f-da55-4f59-bb0c-6b2520237c06:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1467701866
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-5d8c8a7f-da55-4f59-bb0c-6b2520237c06
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:33,869 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:33,869 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,871 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:33,874 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:33,875 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,904 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-c42e4fb4-e4dd-4dda-a0e3-d25ea61df076:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,905 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-c42e4fb4-e4dd-4dda-a0e3-d25ea61df076:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1615421924
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-c42e4fb4-e4dd-4dda-a0e3-d25ea61df076
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:33,912 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:33,912 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,914 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:33,917 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:33,917 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:33,949 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-23d1d7b5-2328-4af6-a613-b03fc77a8ee5:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:33,949 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-23d1d7b5-2328-4af6-a613-b03fc77a8ee5:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:33,970 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:33,971 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,972 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:33,974 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,001 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@40a775f4: startup date [Wed Sep 25 23:42:33 EDT 2013]; root of context hierarchy
23:42:34,001 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@47487fd9: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:34,004 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:34,026 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:34,044 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:34,074 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:34,075 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@7bd5ae33: startup date [Wed Sep 25 23:42:34 EDT 2013]; root of context hierarchy
23:42:34,092 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:34,098 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:34,109 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4bdb3aa6: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:34,152 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:34,153 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:34,169 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,192 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:34,194 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:34,240 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-02022d4e-59a0-4f94-a50a-cc45dda4e541:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:34,241 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-02022d4e-59a0-4f94-a50a-cc45dda4e541:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@374635292
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-02022d4e-59a0-4f94-a50a-cc45dda4e541
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:34,249 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:34,249 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:34,251 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:34,253 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:34,254 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:34,295 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-f1d7a348-6dba-4079-aa76-cb426dd01a45:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:34,295 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-f1d7a348-6dba-4079-aa76-cb426dd01a45:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1391047049
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-f1d7a348-6dba-4079-aa76-cb426dd01a45
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:34,300 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:34,300 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:34,302 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:34,305 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:34,305 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:34,331 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-85b9065b-990f-4066-98ca-9afab469d649:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:34,331 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-85b9065b-990f-4066-98ca-9afab469d649:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:34,338 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:34,338 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,339 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,340 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,351 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@7bd5ae33: startup date [Wed Sep 25 23:42:34 EDT 2013]; root of context hierarchy
23:42:34,351 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4bdb3aa6: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:34,355 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:34,435 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:34,455 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:34,515 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:34,515 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@5828da3d: startup date [Wed Sep 25 23:42:34 EDT 2013]; root of context hierarchy
23:42:34,536 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:34,547 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:34,549 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@6377137e: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:34,565 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:34,565 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:34,571 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,605 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:34,609 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:34,638 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-ea55d73f-2a0f-4e82-a9fb-18ae1e01ace1:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:34,638 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-ea55d73f-2a0f-4e82-a9fb-18ae1e01ace1:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1123506383
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-ea55d73f-2a0f-4e82-a9fb-18ae1e01ace1
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:34,643 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:34,644 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:34,646 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:34,648 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:34,648 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:34,706 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-90931a81-2c70-4136-8a0d-acacb5f508b6:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:34,706 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-90931a81-2c70-4136-8a0d-acacb5f508b6:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1466155219
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-90931a81-2c70-4136-8a0d-acacb5f508b6
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:34,710 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:34,710 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:34,712 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:34,715 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:34,715 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:34,742 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-89812569-0d74-44c9-9510-6392fe38e6da:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:34,742 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-89812569-0d74-44c9-9510-6392fe38e6da:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:34,750 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:34,750 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,751 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,752 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,763 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@5828da3d: startup date [Wed Sep 25 23:42:34 EDT 2013]; root of context hierarchy
23:42:34,763 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@6377137e: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:34,773 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:34,820 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:34,851 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:34,860 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:34,860 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@45d23e41: startup date [Wed Sep 25 23:42:34 EDT 2013]; root of context hierarchy
23:42:34,869 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:34,873 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:34,874 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4db98a2e: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:34,891 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:34,891 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:34,897 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:34,911 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:34,913 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:34,956 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-267ea949-3692-432a-8d24-160728253602:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:34,956 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-267ea949-3692-432a-8d24-160728253602:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1831345293
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-267ea949-3692-432a-8d24-160728253602
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:34,964 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:34,964 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:34,966 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:34,968 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:34,968 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:34,994 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-9873d7f4-0cde-458f-bc89-58f291e934e2:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:34,994 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-9873d7f4-0cde-458f-bc89-58f291e934e2:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@2059324971
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-9873d7f4-0cde-458f-bc89-58f291e934e2
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:34,998 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:34,999 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,001 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:35,005 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,005 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:35,031 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-946f28da-dd98-43ad-88cb-f51fd032c966:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:35,031 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-946f28da-dd98-43ad-88cb-f51fd032c966:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:35,038 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,038 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,039 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,041 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,052 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@45d23e41: startup date [Wed Sep 25 23:42:34 EDT 2013]; root of context hierarchy
23:42:35,052 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4db98a2e: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:35,055 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:35,073 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:35,101 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:35,111 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:35,112 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@6432a099: startup date [Wed Sep 25 23:42:35 EDT 2013]; root of context hierarchy
23:42:35,122 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:35,128 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:35,135 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@34b15bc4: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:35,152 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,152 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:35,158 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,171 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:35,174 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:35,203 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-45204cb8-5b17-41c3-8028-c3b4bcc98f36:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:35,204 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-45204cb8-5b17-41c3-8028-c3b4bcc98f36:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1235265205
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-45204cb8-5b17-41c3-8028-c3b4bcc98f36
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:35,210 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:35,210 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,212 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:35,215 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,215 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:35,240 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-d62332ff-2069-4212-b5ed-3d222aa1b491:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:35,240 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-d62332ff-2069-4212-b5ed-3d222aa1b491:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@118555953
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-d62332ff-2069-4212-b5ed-3d222aa1b491
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:35,245 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:35,245 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,247 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:35,250 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,250 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:35,275 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-e8bbeb99-1d75-4b69-af1b-517e00ca9e02:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:35,275 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-e8bbeb99-1d75-4b69-af1b-517e00ca9e02:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:35,281 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,282 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,283 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,284 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,305 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@6432a099: startup date [Wed Sep 25 23:42:35 EDT 2013]; root of context hierarchy
23:42:35,305 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@34b15bc4: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:35,308 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:35,388 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:35,406 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:35,427 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:35,427 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@6ff1f2a0: startup date [Wed Sep 25 23:42:35 EDT 2013]; root of context hierarchy
23:42:35,443 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:35,451 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:35,453 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@118c7cce: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:35,487 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,487 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:35,495 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,547 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:35,549 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:35,590 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-225856b7-57c2-4095-b3f8-262bac206fce:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:35,590 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-225856b7-57c2-4095-b3f8-262bac206fce:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@977543014
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-225856b7-57c2-4095-b3f8-262bac206fce
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:35,595 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:35,596 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,598 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:35,600 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,600 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:35,625 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-8221cf0c-9989-4d27-8aaf-4ee80e5fc2fe:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:35,625 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-8221cf0c-9989-4d27-8aaf-4ee80e5fc2fe:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1174357210
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-8221cf0c-9989-4d27-8aaf-4ee80e5fc2fe
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:35,629 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:35,629 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,632 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:35,634 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,634 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:35,660 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-df22ccbc-6c8a-42c7-964c-7b275060ec58:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:35,660 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-df22ccbc-6c8a-42c7-964c-7b275060ec58:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:35,666 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,666 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,667 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,668 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,710 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@6ff1f2a0: startup date [Wed Sep 25 23:42:35 EDT 2013]; root of context hierarchy
23:42:35,711 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@118c7cce: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:35,714 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/mapreduce/JarTests-context.xml]
23:42:35,737 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/hadoop-ctx.xml]
23:42:35,751 INFO Test worker xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/data/hadoop/batch-common.xml]
23:42:35,760 INFO Test worker support.DefaultListableBeanFactory - Overriding bean definition for bean 'mainJob': replacing [Generic bean: class [org.springframework.batch.core.configuration.xml.SimpleFlowFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] with [Generic bean: class [org.springframework.batch.core.configuration.xml.JobParserJobFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
23:42:35,760 INFO Test worker support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@7d9d7263: startup date [Wed Sep 25 23:42:35 EDT 2013]; root of context hierarchy
23:42:35,773 INFO Test worker config.PropertyPlaceholderConfigurer - Loading properties file from class path resource [test.properties]
23:42:35,779 INFO Test worker annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
23:42:35,781 INFO Test worker support.DefaultListableBeanFactory - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@f300679: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
23:42:35,802 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,802 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:35,810 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:35,840 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:35,860 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:35,896 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-49e9f0dd-d399-49da-984b-43db307dc013:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:35,896 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-49e9f0dd-d399-49da-984b-43db307dc013:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1135753506
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-49e9f0dd-d399-49da-984b-43db307dc013
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:35,904 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:35,904 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,910 INFO Test worker mapreduce.JarRunner - Invoking [class test.MainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[bad]]
23:42:35,914 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,914 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:35,973 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-cc396d49-47f1-4e03-80bd-35d52fc2b901:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:35,975 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-cc396d49-47f1-4e03-80bd-35d52fc2b901:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
*** New Config is ***Config@1387550948
Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, Custom-cfg-for- class path resource [some-jar.jar]-cc396d49-47f1-4e03-80bd-35d52fc2b901
{mapreduce.job.ubertask.enable=false, yarn.resourcemanager.max-completed-applications=10000, yarn.resourcemanager.delayed.delegation-token.removal-interval-ms=30000, mapreduce.client.submit.file.replication=10, mapreduce.jobhistory.cleaner.interval-ms=86400000, yarn.nodemanager.container-manager.thread-count=20, yarn.nodemanager.pmem-check-enabled=true, dfs.image.transfer.bandwidthPerSec=0, mapreduce.jobhistory.done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done, mapreduce.tasktracker.healthchecker.interval=60000, mapreduce.jobtracker.staging.root.dir=${hadoop.tmp.dir}/mapred/staging, yarn.resourcemanager.recovery.enabled=false, dfs.block.access.token.lifetime=600, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, mapreduce.client.completion.pollinterval=5000, mapreduce.job.ubertask.maxreduces=1, mapreduce.jobhistory.client.thread-count=10, mapreduce.reduce.shuffle.memory.limit.percent=0.25, yarn.client.max-nodemanagers-proxies=500, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.http.authentication.kerberos.keytab=${user.home}/hadoop.keytab, yarn.nodemanager.keytab=/etc/krb5.keytab, io.seqfile.sorter.recordlimit=1000000, mapreduce.task.io.sort.factor=10, s3.blocksize=67108864, yarn.nodemanager.disk-health-checker.interval-ms=120000, yarn.admin.acl=*, mapreduce.job.speculative.speculativecap=0.1, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.delegation.token.renew-interval=86400000, yarn.nodemanager.resource.memory-mb=8192, io.map.index.interval=128, s3.client-write-packet-size=65536, yarn.resourcemanager.fs.state-store.uri=${hadoop.tmp.dir}/yarn/system/rmstore, mapreduce.task.files.preserve.failedtasks=false, dfs.namenode.http-address=0.0.0.0:50070, ha.zookeeper.session-timeout.ms=5000, hadoop.hdfs.configuration.version=1, s3.replication=3, dfs.datanode.balance.bandwidthPerSec=1048576, mapreduce.reduce.shuffle.connect.timeout=180000, hadoop.ssl.enabled=false, dfs.journalnode.rpc-address=0.0.0.0:8485, mapreduce.job.counters.max=120, dfs.datanode.readahead.bytes=4193404, ipc.client.connect.max.retries.on.timeouts=45, mapreduce.job.complete.cancel.delegation.tokens=true, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.client.failover.max.attempts=15, dfs.namenode.replication.work.multiplier.per.iteration=2, fs.trash.interval=0, yarn.resourcemanager.admin.address=${yarn.resourcemanager.hostname}:8033, ha.health-monitor.check-interval.ms=1000, hadoop.jetty.logs.serve.aliases=true, hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LOCALHOST, mapreduce.tasktracker.taskmemorymanager.monitoringinterval=5000, mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.Shuffle, s3native.blocksize=67108864, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, ha.health-monitor.sleep-after-disconnect.ms=1000, yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000, dfs.datanode.http.address=0.0.0.0:50075, dfs.encrypt.data.transfer=false, yarn.log-aggregation.retain-check-interval-seconds=-1, mapreduce.jobtracker.jobhistory.task.numberprogresssplits=12, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.client.use.datanode.hostname=false, mapreduce.map.cpu.vcores=1, yarn.acl.enable=true, hadoop.security.instrumentation.requires.admin=false, yarn.nodemanager.localizer.fetch.thread-count=4, hadoop.security.authorization=false, dfs.namenode.fs-limits.min-block-size=1048576, dfs.client.failover.connection.retries.on.timeouts=0, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.shuffle.max.connections=0, dfs.namenode.safemode.extension=30000, mapreduce.shuffle.port=13562, yarn.log-aggregation-enable=false, mapreduce.reduce.log.level=INFO, dfs.datanode.sync.behind.writes=false, mapreduce.jobtracker.instrumentation=org.apache.hadoop.mapred.JobTrackerMetricsInst, dfs.https.server.keystore.resource=ssl-server.xml, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, dfs.namenode.replication.min=1, s3native.bytes-per-checksum=512, mapreduce.tasktracker.tasks.sleeptimebeforesigkill=5000, tfile.fs.output.buffer.size=262144, yarn.nodemanager.local-dirs=${hadoop.tmp.dir}/nm-local-dir, mapreduce.jobtracker.persist.jobstatus.active=true, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer, dfs.namenode.safemode.min.datanodes=0, mapreduce.tasktracker.local.dir.minspacestart=0, hadoop.security.uid.cache.secs=14400, dfs.client.https.need-auth=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy, dfs.client.https.keystore.resource=ssl-client.xml, dfs.namenode.max.objects=0, hadoop.ssl.client.conf=ssl-client.xml, dfs.namenode.safemode.threshold-pct=0.999f, mapreduce.tasktracker.local.dir.minspacekill=0, mapreduce.jobtracker.retiredjobs.cache.size=1000, dfs.blocksize=134217728, yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler, mapreduce.job.reduce.slowstart.completedmaps=0.05, mapreduce.job.end-notification.retry.attempts=0, mapreduce.tasktracker.outofband.heartbeat=false, io.native.lib.available=true, mapreduce.jobtracker.persist.jobstatus.hours=1, dfs.client-write-packet-size=65536, mapreduce.client.progressmonitor.pollinterval=1000, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.ha.log-roll.period=120, mapreduce.reduce.input.buffer.percent=0.0, mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.DefaultCodec, mapreduce.map.skip.proc.count.autoincr=true, dfs.client.failover.sleep.base.millis=500, dfs.datanode.directoryscan.threads=1, mapreduce.cluster.local.dir=${hadoop.tmp.dir}/mapred/local, mapreduce.jobtracker.address=local, dfs.permissions.enabled=true, mapreduce.tasktracker.taskcontroller=org.apache.hadoop.mapred.DefaultTaskController, mapreduce.reduce.shuffle.parallelcopies=5, dfs.support.append=true, yarn.nodemanager.env-whitelist=JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME, mapreduce.jobtracker.heartbeats.in.second=100, mapreduce.job.maxtaskfailures.per.tracker=3, mapreduce.shuffle.ssl.enabled=false, ipc.client.connection.maxidletime=10000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, tmpfiles=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, dfs.blockreport.intervalMsec=21600000, fs.s3.sleepTimeSeconds=10, dfs.namenode.replication.considerLoad=true, yarn.scheduler.maximum-allocation-vcores=32, dfs.client.block.write.retries=3, hadoop.ssl.server.conf=ssl-server.xml, dfs.namenode.name.dir.restore=false, dfs.datanode.hdfs-blocks-metadata.enabled=false, ha.zookeeper.parent-znode=/hadoop-ha, io.seqfile.lazydecompress=true, yarn.client.app-submission.poll-interval=1000, mapreduce.reduce.merge.inmem.threshold=1000, dfs.https.enable=false, mapreduce.input.fileinputformat.split.minsize=0, dfs.replication=3, ipc.client.tcpnodelay=false, dfs.namenode.accesstime.precision=3600000, s3.stream-buffer-size=4096, mapreduce.jobtracker.tasktracker.maxblacklists=4, mapreduce.job.jvm.numtasks=1, mapreduce.task.io.sort.mb=100, io.file.buffer.size=4096, dfs.namenode.audit.loggers=default, yarn.nodemanager.admin-env=MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX, dfs.namenode.checkpoint.txns=1000000, mapreduce.job.jar=file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar, mapreduce.job.split.metainfo.maxsize=10000000, yarn.resourcemanager.scheduler.monitor.enable=false, mapreduce.reduce.maxattempts=4, rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB=org.apache.hadoop.ipc.ProtobufRpcEngine, yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms=1000, dfs.ha.tail-edits.period=60, fs.s3.buffer.dir=${hadoop.tmp.dir}/s3, hadoop.security.authentication=simple, mapreduce.jobtracker.taskscheduler=org.apache.hadoop.mapred.JobQueueTaskScheduler, yarn.app.mapreduce.am.job.task.listener.thread-count=30, dfs.namenode.avoid.read.stale.datanode=false, mapreduce.job.reduces=1, mapreduce.map.sort.spill.percent=0.80, dfs.client.file-block-storage-locations.timeout=60, dfs.datanode.drop.cache.behind.writes=false, mapreduce.job.end-notification.retry.interval=1000, mapreduce.job.maps=2, mapreduce.job.speculative.slownodethreshold=1.0, mapreduce.jobhistory.minicluster.fixed.ports=false, dfs.block.access.token.enable=false, tfile.fs.input.buffer.size=262144, mapreduce.map.speculative=true, dfs.journalnode.http-address=0.0.0.0:8480, mapreduce.job.acl-view-job= , mapreduce.reduce.shuffle.retry-delay.max.ms=60000, mapreduce.job.end-notification.max.retry.interval=5000, yarn.ipc.serializer.type=protocolbuffers, mapreduce.tasktracker.http.threads=40, ftp.blocksize=67108864, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, ha.failover-controller.cli-check.rpc-timeout.ms=20000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.replication.interval=3, yarn.nodemanager.resourcemanager.connect.wait.secs=900, mapreduce.task.skip.start.attempts=2, dfs.namenode.https-address=0.0.0.0:50470, mapreduce.jobtracker.persist.jobstatus.dir=/jobtracker/jobsInfo, dfs.ha.automatic-failover.enabled=false, ipc.client.kill.max=10, yarn.nodemanager.linux-container-executor.cgroups.mount=false, mapreduce.jobhistory.keytab=/etc/security/keytab/jhs.service.keytab, dfs.image.transfer.timeout=600000, yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/hadoop-yarn, dfs.client.failover.sleep.max.millis=15000, mapreduce.job.end-notification.max.attempts=5, mapreduce.jobhistory.max-age-ms=604800000, mapreduce.task.tmp.dir=./tmp, dfs.default.chunk.view.size=32768, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, dfs.datanode.failed.volumes.tolerated=0, hadoop.http.authentication.type=simple, dfs.datanode.data.dir.perm=700, yarn.resourcemanager.client.thread-count=50, ipc.server.listen.queue.size=128, mapreduce.reduce.skip.maxgroups=0, file.stream-buffer-size=4096, dfs.namenode.fs-limits.max-directory-items=0, yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore, io.mapfile.bloom.size=1048576, yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor, mapreduce.map.maxattempts=4, mapreduce.jobtracker.jobhistory.block.size=3145728, yarn.log-aggregation.retain-seconds=-1, ftp.replication=3, yarn.app.mapreduce.am.job.committer.cancel-timeout=60000, mapreduce.jobtracker.http.address=0.0.0.0:50030, mapreduce.jobhistory.intermediate-done-dir=${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate, yarn.nodemanager.health-checker.script.timeout-ms=1200000, mapreduce.jobhistory.address=0.0.0.0:10020, mapreduce.jobtracker.taskcache.levels=2, dfs.datanode.dns.nameserver=default, mapreduce.application.classpath=$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*, yarn.nodemanager.log.retain-seconds=10800, yarn.nodemanager.local-cache.max-files-per-directory=8192, mapred.child.java.opts=-Xmx200m, dfs.replication.max=512, map.sort.class=org.apache.hadoop.util.QuickSort, dfs.stream-buffer-size=4096, dfs.namenode.backup.address=0.0.0.0:50100, hadoop.util.hash.type=murmur, dfs.block.access.key.update.interval=600, mapreduce.jobhistory.move.interval-ms=180000, dfs.datanode.use.datanode.hostname=false, mapreduce.reduce.skip.proc.count.autoincr=true, dfs.datanode.dns.interface=default, dfs.namenode.backup.http-address=0.0.0.0:50105, yarn.nodemanager.container-monitor.interval-ms=3000, yarn.client.nodemanager-client-async.thread-pool-max-size=500, yarn.nodemanager.disk-health-checker.min-healthy-disks=0.25, ha.zookeeper.acl=world:anyone:rwcda, dfs.namenode.enable.retrycache=true, yarn.nodemanager.sleep-delay-before-sigkill.ms=250, io.map.index.skip=0, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, dfs.namenode.logging.level=info, fs.s3.maxRetries=4, ha.failover-controller.new-active.rpc-timeout.ms=60000, s3native.client-write-packet-size=65536, hadoop.http.staticuser.user=dr.who, yarn.resourcemanager.amliveliness-monitor.interval-ms=1000, mapreduce.reduce.speculative=true, mapreduce.client.output.filter=FAILED, mapreduce.jobhistory.datestring.cache.size=200000, mapreduce.ifile.readahead.bytes=4194304, mapreduce.tasktracker.report.address=127.0.0.1:0, mapreduce.tasktracker.map.tasks.maximum=2, mapreduce.task.userlog.limit.kb=0, hadoop.http.authentication.simple.anonymous.allowed=true, hadoop.fuse.timer.period=5, dfs.namenode.num.extra.edits.retained=1000000, mapreduce.job.classloader.system.classes=java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop., hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, yarn.nodemanager.resourcemanager.connect.retry_interval.secs=30, dfs.namenode.handler.count=10, yarn.resourcemanager.connect.max-wait.ms=900000, fs.automatic.close=true, mapreduce.tasktracker.healthchecker.script.timeout=600000, dfs.datanode.directoryscan.interval=21600, yarn.resourcemanager.address=sandbox:8050, yarn.nodemanager.health-checker.interval-ms=600000, dfs.client.file-block-storage-locations.num-threads=10, yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs=86400, mapreduce.reduce.markreset.buffer.percent=0.0, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, mapreduce.map.log.level=INFO, dfs.bytes-per-checksum=512, yarn.nodemanager.localizer.address=${yarn.nodemanager.hostname}:8040, dfs.namenode.checkpoint.max-retries=3, yarn.resourcemanager.keytab=/etc/krb5.keytab, ftp.stream-buffer-size=4096, ha.health-monitor.rpc-timeout.ms=45000, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.edits.noeditlogchannelflush=false, hadoop.security.group.mapping.ldap.search.attr.member=member, dfs.blockreport.initialDelay=0, mapreduce.job.classloader=false, yarn.nm.liveness-monitor.expiry-interval-ms=600000, io.compression.codec.bzip2.library=system-native, hadoop.http.authentication.token.validity=36000, dfs.namenode.delegation.token.max-lifetime=604800000, mapreduce.job.hdfs-servers=${fs.defaultFS}, s3native.replication=3, dfs.heartbeat.interval=3, yarn.nodemanager.localizer.client.thread-count=5, yarn.resourcemanager.container.liveness-monitor.interval-ms=600000, dfs.ha.fencing.ssh.connect-timeout=30000, yarn.am.liveness-monitor.expiry-interval-ms=600000, net.topology.impl=org.apache.hadoop.net.NetworkTopology, mapreduce.task.profile=false, web=war, yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler, mapreduce.tasktracker.instrumentation=org.apache.hadoop.mapred.TaskTrackerMetricsInst, mapreduce.tasktracker.http.address=0.0.0.0:50060, mapreduce.jobhistory.webapp.address=0.0.0.0:19888, yarn.ipc.rpc.class=org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, cfg=main, yarn.resourcemanager.application-tokens.master-key-rolling-interval-secs=86400, yarn.resourcemanager.am.max-attempts=2, mapreduce.job.ubertask.maxmaps=9, yarn.scheduler.maximum-allocation-mb=8192, mapreduce.job.userlog.retain.hours=24, dfs.namenode.secondary.http-address=0.0.0.0:50090, mapreduce.task.timeout=600000, mapreduce.framework.name=local, mapreduce.jobhistory.loadedjobs.cache.size=5, ipc.client.idlethreshold=4000, ftp.bytes-per-checksum=512, ipc.server.tcpnodelay=false, dfs.namenode.stale.datanode.interval=30000, yarn.resourcemanager.hostname=0.0.0.0, s3.bytes-per-checksum=512, mapreduce.job.speculative.slowtaskthreshold=1.0, yarn.nodemanager.localizer.cache.target-size-mb=10240, yarn.nodemanager.remote-app-log-dir=/tmp/logs, fs.s3.block.size=67108864, mapreduce.job.queuename=default, dfs.client.failover.connection.retries=0, yarn.scheduler.minimum-allocation-mb=1024, hadoop.rpc.protection=authentication, dfs.namenode.retrycache.expirytime.millis=600000, yarn.app.mapreduce.client-am.ipc.max-retries=1, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, ftp.client-write-packet-size=65536, yarn.nodemanager.address=${yarn.nodemanager.hostname}:0, fs.defaultFS=hdfs://sandbox:8020, mapreduce.task.merge.progress.records=10000, yarn.resourcemanager.scheduler.client.thread-count=50, file.client-write-packet-size=65536, mapreduce.reduce.cpu.vcores=1, yarn.nodemanager.delete.thread-count=4, yarn.resourcemanager.scheduler.address=${yarn.resourcemanager.hostname}:8030, fs.trash.checkpoint.interval=0, hadoop.http.authentication.signature.secret.file=${user.home}/hadoop-http-auth-signature-secret, s3native.stream-buffer-size=4096, mapreduce.reduce.shuffle.read.timeout=180000, yarn.app.mapreduce.am.command-opts=-Xmx1024m, mapreduce.admin.user.env=LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, mapreduce.local.clientfactory.class.name=org.apache.hadoop.mapred.LocalClientFactory, fs.permissions.umask-mode=022, mapreduce.jobhistory.move.thread-count=3, hadoop.common.configuration.version=0.23.0, mapreduce.tasktracker.dns.interface=default, yarn.resourcemanager.connect.retry-interval.ms=30000, mapreduce.output.fileoutputformat.compress.type=RECORD, hadoop.security.group.mapping.ldap.ssl=false, mapreduce.ifile.readahead=true, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, yarn.nodemanager.aux-services.mapreduce.shuffle.class=org.apache.hadoop.mapred.ShuffleHandler, fs.df.interval=60000, mapreduce.reduce.shuffle.input.buffer.percent=0.70, io.seqfile.compress.blocksize=1000000, ipc.client.connect.max.retries=10, hadoop.security.groups.cache.secs=300, dfs.namenode.delegation.key.update-interval=86400000, yarn.nodemanager.vmem-check-enabled=true, yarn.nodemanager.process-kill-wait.ms=2000, yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*, yarn.app.mapreduce.client.max-retries=3, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, yarn.nodemanager.log-aggregation.compression-type=none, hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), dfs.image.compress=false, yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000, yarn.nodemanager.log-dirs=${yarn.log.dir}/userlogs, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, fs.s3n.block.size=67108864, fs.ftp.host=0.0.0.0, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, yarn.app.mapreduce.am.resource.cpu-vcores=1, dfs.datanode.address=0.0.0.0:50010, mapreduce.jobhistory.cleaner.enable=true, mapreduce.map.skip.maxrecords=0, dfs.datanode.https.address=0.0.0.0:50475, yarn.scheduler.minimum-allocation-vcores=1, file.replication=1, yarn.resourcemanager.resource-tracker.address=${yarn.resourcemanager.hostname}:8031, dfs.datanode.drop.cache.behind.reads=false, hadoop.fuse.connection.timeout=300, hadoop.work.around.non.threadsafe.getpwuid=false, mapreduce.jobtracker.restart.recover=false, mapreduce.client.genericoptionsparser.used=true, hadoop.tmp.dir=/tmp/hadoop-${user.name}, mapreduce.tasktracker.indexcache.mb=10, mapreduce.output.fileoutputformat.compress=false, yarn.nodemanager.resource.cpu-vcores=8, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, mapreduce.job.committer.setup.cleanup.needed=true, hadoop.kerberos.kinit.command=kinit, dfs.namenode.fs-limits.max-blocks-per-file=1048576, dfs.webhdfs.enabled=false, dfs.datanode.du.reserved=0, mapreduce.task.profile.reduces=0-2, file.bytes-per-checksum=512, dfs.client.block.write.replace-datanode-on-failure.enable=true, mapreduce.jobtracker.handler.count=10, yarn.app.mapreduce.am.job.committer.commit-window=10000, someparam=somevalue, net.topology.script.number.args=100, mapreduce.task.profile.maps=0-2, dfs.namenode.decommission.interval=30, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, yarn.resourcemanager.webapp.address=${yarn.resourcemanager.hostname}:8088, mapreduce.jobtracker.system.dir=${hadoop.tmp.dir}/mapred/system, yarn.nodemanager.vmem-pmem-ratio=2.1, dfs.namenode.support.allow.format=true, hadoop.ssl.hostname.verifier=DEFAULT, yarn.nodemanager.hostname=0.0.0.0, ipc.client.connect.timeout=20000, mapreduce.jobhistory.principal=jhs/[email protected], io.mapfile.bloom.error.rate=0.005, mapreduce.shuffle.ssl.file.buffer.size=65536, dfs.permissions.superusergroup=supergroup, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, mapreduce.jobtracker.expire.trackers.interval=600000, mapreduce.cluster.acls.enabled=false, yarn.nodemanager.remote-app-log-dir-suffix=logs, ha.failover-controller.graceful-fence.connection.retries=1, ha.health-monitor.connect-retry-interval.ms=1000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, mapreduce.reduce.shuffle.merge.percent=0.66, dfs.namenode.checkpoint.check.period=60, yarn.app.mapreduce.am.resource.mb=1536, tfile.io.chunk.size=1048576, mapreduce.jobtracker.maxtasks.perjob=-1, file.blocksize=67108864, mapreduce.jobtracker.jobhistory.lru.cache.size=5, yarn.resourcemanager.nm.liveness-monitor.interval-ms=1000, mapreduce.job.acl-modify-job= , yarn.nodemanager.webapp.address=${yarn.nodemanager.hostname}:8042, mapreduce.tasktracker.reduce.tasks.maximum=2, mapreduce.am.max-attempts=2, mapreduce.cluster.temp.dir=${hadoop.tmp.dir}/mapred/temp, io.skip.checksum.errors=false, mapreduce.jobhistory.joblist.cache.size=20000, yarn.app.mapreduce.am.staging-dir=/tmp/hadoop-yarn/staging, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.datanode.handler.count=10, dfs.namenode.decommission.nodes.per.interval=5, fs.ftp.host.port=21, dfs.namenode.fs-limits.max-component-length=0, yarn.resourcemanager.admin.client.thread-count=1, dfs.namenode.checkpoint.period=3600, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, dfs.namenode.retrycache.heap.percent=0.03f, yarn.resourcemanager.resource-tracker.client.thread-count=50, mapreduce.tasktracker.dns.nameserver=default, ipc.client.fallback-to-simple-auth-allowed=false, mapreduce.map.output.compress=false, dfs.datanode.ipc.address=0.0.0.0:50020, yarn.nodemanager.delete.debug-delay-sec=0, hadoop.ssl.require.client.cert=false, dfs.datanode.max.transfer.threads=4096}
*** Received args ***[bad]
23:42:35,983 WARN Test worker conf.Configuration - mapred.jar is deprecated. Instead, use mapreduce.job.jar
23:42:35,984 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:35,992 INFO Test worker mapreduce.JarRunner - Invoking [class test.OtherMainClass] from jar [file:/Users/trisberg/Projects/spring-hadoop-1.0.x/build/resources/test/some-jar.jar] with args [[42]]
23:42:35,996 WARN Test worker conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
23:42:35,997 WARN Test worker conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
23:42:36,041 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-3eea5403-5297-4d56-b6c2-466cf90cb19b:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
23:42:36,042 WARN Test worker conf.Configuration - Custom-cfg-for- class path resource [some-jar.jar]-3eea5403-5297-4d56-b6c2-466cf90cb19b:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
23:42:36,049 WARN Test worker conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
23:42:36,050 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:36,057 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:36,058 INFO Test worker fs.TrashPolicyDefault - Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
23:42:36,077 INFO Test worker support.GenericApplicationContext - Closing org.springframework.context.support.GenericApplicationContext@7d9d7263: startup date [Wed Sep 25 23:42:35 EDT 2013]; root of context hierarchy
23:42:36,079 INFO Test worker support.DefaultListableBeanFactory - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@f300679: defining beans [ppc,hadoopFs,hadoopResourceLoader,hadoopConfiguration,cfg-init,fs-init,rl-init,org.springframework.data.hadoop.scripting.HdfsScriptRunner#0,jobRepository,transactionManager,jobLauncher,taskExecutor,bad-main-class,other-class,org.springframework.batch.core.scope.internalStepScope,org.springframework.beans.factory.config.CustomEditorConfigurer,org.springframework.batch.core.configuration.xml.CoreNamespacePostProcessor,tasklet-ns,cleanup,org.springframework.data.hadoop.scripting.HdfsScriptRunner#1,script-tasklet,wordcount,mainJob,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy
Removing incorrect key [org.springframework.data.hadoop.jar.cfg] w/ value Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml
Removing incorrect key [org.springframework.data.hadoop.jar.other.cfg] w/ value Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml
Removing incorrect key [org.springframework.data.hadoop.jar.args] w/ value [Ljava.lang.String;@6f119af2
Removing incorrect key [org.springframework.data.jar.exit.exception] w/ value org.springframework.data.hadoop.mapreduce.ExecutionUtils$ExitTrapped
Removing incorrect key [org.springframework.data.hadoop.jar.other.args] w/ value [Ljava.lang.String;@6cd3bed5