Friday, May 22, 2015

, ,

Could not find any valid local directory for jobcache EXCEPTION

Chris,

Thank you very much. It does help, a lot. I had a feeling that it's
something like that.

I wish you all the best,
Marko

Quoting Chris Nauroth :

> Based on this stack trace, I'm guessing that you're running a 1.x version
> of Hadoop.
>
> The TaskTracker uses a set of local directories on the node for storage of
> submitted job files during the task's execution. This is configured in
> mapred-site.xml in the property named mapred.job.local.dir. The
> DiskErrorException means that even after trying all directories configured
> in mapped.job.local.dir, the TaskTracker couldn't find a place to store
> the files. Possible root causes are misconfiguration, permissions on the
> local directories blocking access, disks are full, or disks have failed
> and gone into read-only mode.
>
> I hope this helps.
>
> --Chris Nauroth
>
>
>
>
> On 5/21/15, 3:51 AM, "Marko Dinic" wrote:
>
>> I'm new to Hadoop and I'm getting the following exception when I try to
>> run my job on Hadoop cluster:
>>
>> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
>> any valid local directory for jobcache/job_201409031055_3865/jars/job.jar
>> at
>> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathFor
>> Write(LocalDirAllocator.java:376)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca
>> tor.java:146)
>> at
>> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAlloca
>> tor.java:127)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java
>> :268)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:3
>> 80)
>> at
>> org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:3
>> 70)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskCo
>> ntroller.java:232)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1381)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java
>>
>> Can anyone please tell me what seems to be the problem?
>>
>> Best regards,
>> Marko

Thursday, May 21, 2015

, , ,

SparkPi fails with Job aborted due to stage failure: Task serialization failed:

Thank you Rohith for the reply and I wasn’t exactly sure what to make of those log entries.  I had assumed that from Yarn’s perspective, yes the container was setup and broken down completely.  I’ll send this to the Spark user mailing list since it does make better sense to be in that forum.

Regards,
Keith

From: Rohith Sharma K S 
Sent: Wednesday, May 20, 2015 2:06 AM

Hi,

From the ResourceManager log, it is very clear that Job has succeeded. There is no problem running Spark Applications.

2015-05-19 19:53:20,388 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=Application Finished - Succeeded      TARGET=RMAppManager     RESULT=SUCCESS     APPID=application_1432064564266_0003
2015-05-19 19:53:20,389 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1432064564266_0003,name=Spark Pi,user=testuser,queue=default,state=FINISHED,trackingUrl=https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003/A,appMasterHost=ip-10-10-128-10.ec2.internal,startTime=1432065172758,finishTime=1432065198967,finalStatus=SUCCEEDED

But I do not have idea what does spark is doing internally.  Would you mind sending mail to spark user mailing lists.


Thanks & Regards
Rohith Sharma K S

From: Nance, Keith 
Sent: 20 May 2015 02:05

All, unable to find any reference to my issue with spark.  Any ideas?  Thanks for any and all help
Attached are logs from the Spark job (SparkPi) results, Userlog, Nodemanager, and Resourcemanager.
###: SPARK JOB RESULTS :###
###########################
[testuser@ip-10-10-127-10 spark]$ ./bin/spark-submit --verbose --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 1 --executor-cores 1 lib/spark-examples*.jar 10
/home/testuser/spark/conf/spark-env.sh: line 55: -Dspark.history.kerberos.principal=spark/ip-10-10-127-10.ec2.internal@MALARD.LOCAL: No such file or directory
Using properties file: /home/testuser/spark/conf/spark-defaults.conf
Adding default property: spark.broadcast.port=8004
Adding default property: spark.executor.extraJavaOptions=-XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
Adding default property: spark.akka.threads=1
Adding default property: spark.ui.port=4040
Adding default property: spark.driver.port=8001
Adding default property: spark.akka.heartbeat.interval=100
Adding default property: spark.yarn.access.namenodes=hdfs://10.10.10.10:8020
Adding default property: spark.executor.port=8002
Adding default property: spark.logConf=true
Adding default property: spark.yarn.am.extraJavaOptions=-Djava.net.preferIPv4Stack=true
Adding default property: spark.worker.ui.port=8081
Adding default property: spark.replClassServer.port=8006
Adding default property: spark.eventlog.dir=hdfs://10.10.10.10:8020/user/testuser/spark/eventlog
Adding default property: spark.driver.extraJavaOptions=-Djava.net.preferIPv4Stack=true
Adding default property: spark.blockManager.port=8007
Adding default property: spark.yarn.am.waitTime=200000
Adding default property: spark.master=yarn-client
Adding default property: spark.yarn.preserve.staging.files=true
Adding default property: spark.fileserver.port=8003
Adding default property: spark.authenticate=true
Adding default property: spark.yarn.am.port=8008
Adding default property: spark.authenticate.secret=fubar
Adding default property: spark.master.ui.port=8080
Adding default property: spark.history.ui.port=18080
Adding default property: spark.eventlog.enabled=true
Parsed arguments:
  master                  yarn-client
  deployMode              null
  executorMemory          null
  executorCores           1
  totalExecutorCores      null
  propertiesFile          /home/testuser/spark/conf/spark-defaults.conf
  driverMemory            null
  driverCores             null
  driverExtraClassPath    null
  driverExtraLibraryPath  null
 driverExtraJavaOptions  -Djava.net.preferIPv4Stack=true
  supervise               false
  queue                   null
  numExecutors            1
  files                   null
  pyFiles                 null
  archives                null
  mainClass               org.apache.spark.examples.SparkPi
  primaryResource         file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar
  name                    org.apache.spark.examples.SparkPi
  childArgs               [10]
  jars                    null
  packages                null
  repositories            null
  verbose                 true

Spark properties used, including those specified through
--conf and those from the properties file /home/testuser/spark/conf/spark-defaults.conf:
  spark.blockManager.port -> 8007
  spark.yarn.access.namenodes -> hdfs://10.10.10.10:8020
  spark.broadcast.port -> 8004
  spark.authenticate.secret -> fubar
  spark.authenticate -> true
  spark.eventlog.dir -> hdfs://10.10.10.10:8020/user/testuser/spark/eventlog
  spark.logConf -> true
  spark.replClassServer.port -> 8006
  spark.history.ui.port -> 18080
  spark.fileserver.port -> 8003
  spark.ui.port -> 4040
  spark.executor.extraJavaOptions -> -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
  spark.driver.extraJavaOptions -> -Djava.net.preferIPv4Stack=true
  spark.yarn.am.waitTime -> 200000
  spark.yarn.am.extraJavaOptions -> -Djava.net.preferIPv4Stack=true
  spark.worker.ui.port -> 8081
  spark.driver.port -> 8001
  spark.master -> yarn-client
  spark.yarn.preserve.staging.files -> true
  spark.yarn.am.port -> 8008
  spark.akka.heartbeat.interval -> 100
  spark.executor.port -> 8002
  spark.master.ui.port -> 8080
  spark.eventlog.enabled -> true
  spark.akka.threads -> 1


Main class:
org.apache.spark.examples.SparkPi
Arguments:
10
System properties:
spark.blockManager.port -> 8007
spark.yarn.access.namenodes -> hdfs://10.10.10.10:8020
spark.broadcast.port -> 8004
spark.authenticate.secret -> fubar
spark.eventlog.dir -> hdfs://10.10.10.10:8020/user/testuser/spark/eventlog
spark.authenticate -> true
spark.executor.instances -> 1
spark.logConf -> true
spark.replClassServer.port -> 8006
spark.history.ui.port -> 18080
spark.fileserver.port -> 8003
SPARK_SUBMIT -> true
spark.ui.port -> 4040
spark.executor.extraJavaOptions -> -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
spark.app.name -> org.apache.spark.examples.SparkPi
spark.driver.extraJavaOptions -> -Djava.net.preferIPv4Stack=true
spark.jars -> file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar
spark.yarn.am.waitTime -> 200000
spark.yarn.am.extraJavaOptions -> -Djava.net.preferIPv4Stack=true
spark.master -> yarn-client
spark.worker.ui.port -> 8081
spark.driver.port -> 8001
spark.yarn.preserve.staging.files -> true
spark.yarn.am.port -> 8008
spark.akka.heartbeat.interval -> 100
spark.executor.port -> 8002
spark.executor.cores -> 1
spark.eventlog.enabled -> true
spark.master.ui.port -> 8080
spark.akka.threads -> 1
Classpath elements:
file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar


15/05/19 19:52:37 INFO spark.SparkContext: Running Spark version 1.3.1
15/05/19 19:52:37 INFO spark.SparkContext: Spark configuration:
spark.akka.heartbeat.interval=100
spark.akka.threads=1
spark.app.name=Spark Pi
spark.authenticate=true
spark.authenticate.secret=fubar
spark.blockManager.port=8007
spark.broadcast.port=8004
spark.driver.extraJavaOptions=-Djava.net.preferIPv4Stack=true
spark.driver.port=8001
spark.eventlog.dir=hdfs://10.10.10.10:8020/user/testuser/spark/eventlog
spark.eventlog.enabled=true
spark.executor.cores=1
spark.executor.extraJavaOptions=-XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
spark.executor.instances=1
spark.executor.port=8002
spark.fileserver.port=8003
spark.history.ui.port=18080
spark.jars=file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar
spark.logConf=true
spark.master=yarn-client
spark.master.ui.port=8080
spark.replClassServer.port=8006
spark.ui.port=4040
spark.worker.ui.port=8081
spark.yarn.access.namenodes=hdfs://10.10.10.10:8020
spark.yarn.am.extraJavaOptions=-Djava.net.preferIPv4Stack=true
spark.yarn.am.port=8008
spark.yarn.am.waitTime=200000
spark.yarn.preserve.staging.files=true
15/05/19 19:52:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/05/19 19:52:39 INFO spark.SecurityManager: Changing view acls to: testuser
15/05/19 19:52:39 INFO spark.SecurityManager: Changing modify acls to: testuser
15/05/19 19:52:39 INFO spark.SecurityManager: adding secret to credentials in yarn mode
15/05/19 19:52:39 INFO spark.SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(testuser); users with modify permissions: Set(testuser)
15/05/19 19:52:40 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/05/19 19:52:41 INFO Remoting: Starting remoting
15/05/19 19:52:41 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001]
15/05/19 19:52:41 INFO util.Utils: Successfully started service 'sparkDriver' on port 8001.
15/05/19 19:52:41 INFO spark.SparkEnv: Registering MapOutputTracker
15/05/19 19:52:41 INFO spark.SparkEnv: Registering BlockManagerMaster
15/05/19 19:52:41 INFO storage.DiskBlockManager: Created local directory at /scratch/spark-17902ac9-b400-4698-97c3-069d804a29e3/blockmgr-d95b3bdf-9c4d-4b48-97a5-4983dd2ab66d
15/05/19 19:52:41 INFO storage.MemoryStore: MemoryStore started with capacity 267.3 MB
15/05/19 19:52:42 INFO spark.HttpFileServer: HTTP File server directory is /scratch/spark-d2fb7948-5ab3-4f22-804c-7485d209bd3e/httpd-91938877-b371-4c0c-ba3d-bb7ee8ec4e09
15/05/19 19:52:42 INFO spark.HttpServer: Starting HTTP Server
15/05/19 19:52:42 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/05/19 19:52:42 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:8003
15/05/19 19:52:42 INFO util.Utils: Successfully started service 'HTTP file server' on port 8003.
15/05/19 19:52:42 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/05/19 19:52:42 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/05/19 19:52:42 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/05/19 19:52:42 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
15/05/19 19:52:42 INFO ui.SparkUI: Started SparkUI at http://ip-10-10-127-10.ec2.internal:4040
15/05/19 19:52:43 INFO spark.SparkContext: Added JAR file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar at http://10.10.127.10:8003/jars/spark-examples-1.3.1-hadoop2.6.0.jar with timestamp 1432065163724
15/05/19 19:52:44 INFO client.RMProxy: Connecting to ResourceManager at ip-10-10-127-10.ec2.internal/10.10.127.10:8032
15/05/19 19:52:45 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
15/05/19 19:52:45 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
15/05/19 19:52:45 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
15/05/19 19:52:45 INFO yarn.Client: Setting up container launch context for our AM
15/05/19 19:52:45 INFO yarn.Client: Preparing resources for our AM container
15/05/19 19:52:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 5 for testuser on 10.10.10.10:8020
15/05/19 19:52:46 INFO yarn.Client: Uploading resource file:/home/testuser/spark/lib/spark-assembly-1.3.1-hadoop2.6.0.jar -> hdfs://10.10.10.10:8020/user/testuser/.sparkStaging/application_1432064564266_0003/spark-assembly-1.3.1-hadoop2.6.0.jar
15/05/19 19:52:52 INFO yarn.Client: Setting up the launch environment for our AM container
15/05/19 19:52:52 INFO spark.SecurityManager: Changing view acls to: testuser
15/05/19 19:52:52 INFO spark.SecurityManager: Changing modify acls to: testuser
15/05/19 19:52:52 INFO spark.SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(testuser); users with modify permissions: Set(testuser)
15/05/19 19:52:52 INFO yarn.Client: Submitting application 3 to ResourceManager
15/05/19 19:52:53 INFO impl.YarnClientImpl: Submitted application application_1432064564266_0003
15/05/19 19:52:54 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:52:54 INFO yarn.Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1432065172758
         final status: UNDEFINED
         user: testuser
15/05/19 19:52:55 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:52:56 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:52:57 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:52:58 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:52:59 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:00 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:01 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:02 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:03 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:04 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:05 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:06 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:07 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:08 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:09 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:10 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:11 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:12 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:13 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:14 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:15 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:16 INFO yarn.Client: Application report for application_1432064564266_0003 (state: ACCEPTED)
15/05/19 19:53:17 INFO cluster.YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka.tcp://sparkYarnAM@ip-10-10-128-10.ec2.internal:56977/user/YarnAM#-1453228800]
15/05/19 19:53:17 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> ip-10-10-127-10.ec2.internal, PROXY_URI_BASES -> https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003), /proxy/application_1432064564266_0003
15/05/19 19:53:17 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/05/19 19:53:17 INFO yarn.Client: Application report for application_1432064564266_0003 (state: RUNNING)
15/05/19 19:53:17 INFO yarn.Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: ip-10-10-128-10.ec2.internal
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1432065172758
         final status: UNDEFINED
         user: testuser
15/05/19 19:53:17 INFO cluster.YarnClientSchedulerBackend: Application application_1432064564266_0003 has started running.
15/05/19 19:53:17 INFO netty.NettyBlockTransferService: Server created on 8007
15/05/19 19:53:17 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/05/19 19:53:17 INFO storage.BlockManagerMasterActor: Registering block manager ip-10-10-127-10.ec2.internal:8007 with 267.3 MB RAM, BlockManagerId(<driver>, ip-10-10-127-10.ec2.internal, 8007)
15/05/19 19:53:17 INFO storage.BlockManagerMaster: Registered BlockManager
15/05/19 19:53:18 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
15/05/19 19:53:18 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:35
15/05/19 19:53:18 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:35) with 10 output partitions (allowLocal=false)
15/05/19 19:53:18 INFO scheduler.DAGScheduler: Final stage: Stage 0(reduce at SparkPi.scala:35)
15/05/19 19:53:18 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/05/19 19:53:18 INFO scheduler.DAGScheduler: Missing parents: List()
15/05/19 19:53:18 INFO scheduler.DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:31), which has no missing parents
15/05/19 19:53:18 INFO cluster.YarnScheduler: Cancelling stage 0
15/05/19 19:53:18 INFO scheduler.DAGScheduler: Stage 0 (reduce at SparkPi.scala:35) failed in Unknown s
15/05/19 19:53:18 INFO scheduler.DAGScheduler: Job 0 failed: reduce at SparkPi.scala:35, took 0.258029 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.reflect.InvocationTargetException
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:526)
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:68)
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:79)
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
org.apache.spark.SparkContext.broadcast(SparkContext.scala:1051)
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:839)
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:778)
org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:762)
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1362)
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192)
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:847)
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:778)
        at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:762)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1362)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
[testuser@ip-10-10-127-10 spark]$



###: SPARK JOB USERLOG RESULTS :###
###################################
[yarn@ip-10-10-128-10 container_1432064564266_0003_01_000001]$ cat stderr
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/var/hadoop/tmp/yarn/nm-local-dir/usercache/testuser/filecache/13/spark-assembly-1.3.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/var/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/05/19 19:53:09 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
15/05/19 19:53:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/05/19 19:53:12 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1432064564266_0003_000001
15/05/19 19:53:14 INFO spark.SecurityManager: Changing view acls to: testuser
15/05/19 19:53:14 INFO spark.SecurityManager: Changing modify acls to: testuser
15/05/19 19:53:14 INFO spark.SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(testuser); users with modify permissions: Set(testuser)
15/05/19 19:53:15 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/05/19 19:53:15 INFO Remoting: Starting remoting
15/05/19 19:53:16 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkYarnAM@ip-10-10-128-10.ec2.internal:56977]
15/05/19 19:53:16 INFO util.Utils: Successfully started service 'sparkYarnAM' on port 56977.
15/05/19 19:53:16 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
15/05/19 19:53:16 INFO yarn.ApplicationMaster: Driver now available: ip-10-10-127-10.ec2.internal:8001
15/05/19 19:53:16 INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001/user/YarnScheduler
15/05/19 19:53:16 INFO yarn.ApplicationMaster: Add WebUI Filter. AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> ip-10-10-127-10.ec2.internal, PROXY_URI_BASES -> https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003),/proxy/application_1432064564266_0003)
15/05/19 19:53:16 INFO client.RMProxy: Connecting to ResourceManager at /10.10.127.10:8030
15/05/19 19:53:16 INFO yarn.YarnRMClient: Registering the ApplicationMaster
15/05/19 19:53:17 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead
15/05/19 19:53:17 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
15/05/19 19:53:17 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
15/05/19 19:53:18 INFO yarn.ApplicationMaster: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@ip-10-10-128-10.ec2.internal:56977] -> [akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001]
15/05/19 19:53:18 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
15/05/19 19:53:18 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/05/19 19:53:18 INFO yarn.ApplicationMaster: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@ip-10-10-128-10.ec2.internal:56977] -> [akka.tcp://sparkDriver@ip-10-10-127-10.ec2.internal:8001]
15/05/19 19:53:18 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
15/05/19 19:53:18 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
[yarn@ip-10-10-128-10 container_1432064564266_0003_01_000001]$



###: YARN NODEMANAGER LOG RESULTS :###
######################################
2015-05-19 19:52:53,729 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1432064564266_0003_000001 (auth:SIMPLE)
2015-05-19 19:52:53,746 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user appattempt_1432064564266_0003_000001: id: appattempt_1432064564266_0003_000001: no such user

2015-05-19 19:52:53,746 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user appattempt_1432064564266_0003_000001
2015-05-19 19:52:53,746 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for appattempt_1432064564266_0003_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ContainerManagementProtocolPB
2015-05-19 19:52:53,747 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1432064564266_0003_01_000001 by user testuser
2015-05-19 19:52:53,747 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1432064564266_0003
2015-05-19 19:52:53,747 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=testuser     IP=10.10.127.10 OPERATION=Start Container Request       TARGET=ContainerManageImplRESULT=SUCCESS   APPID=application_1432064564266_0003    CONTAINERID=container_1432064564266_0003_01_000001
2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1432064564266_0003 transitioned from NEW to INITING
2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1432064564266_0003_01_000001 to application application_1432064564266_0003
2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1432064564266_0003 transitioned from INITING to RUNNING
2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from NEW to LOCALIZING
2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1432064564266_0003
2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.10.10.10:8020/user/testuser/.sparkStaging/application_1432064564266_0003/spark-assembly-1.3.1-hadoop2.6.0.jar transitioned from INIT to DOWNLOADING
2015-05-19 19:52:53,748 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1432064564266_0003_01_000001
2015-05-19 19:52:53,763 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /var/hadoop/tmp/yarn/nm-local-dir/nmPrivate/container_1432064564266_0003_01_000001.tokens. Credentials list:
2015-05-19 19:52:58,614 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for testing (auth:SIMPLE)
2015-05-19 19:52:58,685 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for testing (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB
2015-05-19 19:53:08,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.10.10.10:8020/user/testuser/.sparkStaging/application_1432064564266_0003/spark-assembly-1.3.1-hadoop2.6.0.jar(->/var/hadoop/tmp/yarn/nm-local-dir/usercache/testuser/filecache/13/spark-assembly-1.3.1-hadoop2.6.0.jar) transitioned from DOWNLOADING to LOCALIZED
2015-05-19 19:53:08,134 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from LOCALIZING to LOCALIZED
2015-05-19 19:53:08,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from LOCALIZED to RUNNING
2015-05-19 19:53:10,044 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1432064564266_0003_01_000001
2015-05-19 19:53:10,092 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 20620 for container-id container_1432064564266_0003_01_000001: 79.5 MB of 1 GB physical memory used; 1.1 GB of 2.1 GB virtual memory used
2015-05-19 19:53:13,105 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 20620 for container-id container_1432064564266_0003_01_000001: 99.8 MB of 1 GB physical memory used; 1.1 GB of 2.1 GB virtual memory used
2015-05-19 19:53:16,122 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 20620 for container-id container_1432064564266_0003_01_000001: 121.4 MB of 1 GB physical memory used; 1.1 GB of 2.1 GB virtual memory used
2015-05-19 19:53:19,129 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 20620 for container-id container_1432064564266_0003_01_000001: 125.8 MB of 1 GB physical memory used; 1.1 GB of 2.1 GB virtual memory used
2015-05-19 19:53:19,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container container_1432064564266_0003_01_000001 succeeded
2015-05-19 19:53:19,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS
2015-05-19 19:53:19,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1432064564266_0003_01_000001
2015-05-19 19:53:19,457 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=testuser     OPERATION=Container Finished - Succeeded        TARGET=ContainerImpl    RESULT=SUCCESS     APPID=application_1432064564266_0003    CONTAINERID=container_1432064564266_0003_01_000001
2015-05-19 19:53:19,457 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1432064564266_0003_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE
2015-05-19 19:53:19,457 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1432064564266_0003_01_000001 from application application_1432064564266_0003
2015-05-19 19:53:19,458 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1432064564266_0003
2015-05-19 19:53:19,458 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting absolute path : /var/hadoop/tmp/yarn/nm-local-dir/usercache/testuser/appcache/application_1432064564266_0003/container_1432064564266_0003_01_000001
2015-05-19 19:53:20,428 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1432064564266_0003_000001 (auth:SIMPLE)
2015-05-19 19:53:20,432 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user appattempt_1432064564266_0003_000001
2015-05-19 19:53:20,432 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for appattempt_1432064564266_0003_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ContainerManagementProtocolPB
2015-05-19 19:53:20,432 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1432064564266_0003_01_000001
2015-05-19 19:53:20,432 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=testuser     IP=10.10.127.10 OPERATION=Stop Container Request        TARGET=ContainerManageImplRESULT=SUCCESS   APPID=application_1432064564266_0003    CONTAINERID=container_1432064564266_0003_01_000001
2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1432064564266_0003_01_000001]
2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1432064564266_0003 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1432064564266_0003
2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1432064564266_0003 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2015-05-19 19:53:20,443 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1432064564266_0003, with delay of 10800 seconds
2015-05-19 19:53:20,443 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Deleting absolute path : /var/hadoop/tmp/yarn/nm-local-dir/usercache/testuser/appcache/application_1432064564266_0003
2015-05-19 19:53:22,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1432064564266_0003_01_000001
root@ip-10-10-128-10:/var/log/hadoop>


###: YARN RESOURCE MANAGER LOGS :###
####################################
2015-05-19 19:52:45,408 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for testuser@MALARD.LOCAL (auth:KERBEROS)
2015-05-19 19:52:45,447 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for testuser@MALARD.LOCAL (auth:KERBEROS) for protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
2015-05-19 19:52:45,494 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 3
2015-05-19 19:52:52,758 WARN org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 3 is invalid, because it is out of the range [1, 2]. Use the global max attempts instead.
2015-05-19 19:52:52,758 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 3 submitted by user testuser
2015-05-19 19:52:52,758 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser IP=10.10.127.10 OPERATION=Submit Application Request    TARGET=ClientRMService  RESULT=SUCCESS     APPID=application_1432064564266_0003
2015-05-19 19:52:52,803 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: application_1432064564266_0003 found existing hdfs token Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020, Ident: (HDFS_DELEGATION_TOKEN token 5 for testuser)
2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020, Ident: (HDFS_DELEGATION_TOKEN token 5 for testuser);exp=1432151572908], for application_1432064564266_0003
2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020, Ident: (HDFS_DELEGATION_TOKEN token 5 for testuser);exp=1432151572908 in 86399980 ms, appId = application_1432064564266_0003
2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1432064564266_0003
2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from NEW to NEW_SAVING
2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1432064564266_0003
2015-05-19 19:52:52,928 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from NEW_SAVING to SUBMITTED
2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1432064564266_0003 user: testuser leaf-queue of parent: root #applications: 1
2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1432064564266_0003 from user: testuser, in queue: default
2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from SUBMITTED to ACCEPTED
2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1432064564266_0003_000001
2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from NEW to SUBMITTED
2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1432064564266_0003 from user: testuser activated in queue: default
2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1432064564266_0003 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@72fa0d32, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1
2015-05-19 19:52:52,929 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1432064564266_0003_000001 to scheduler from user testuser in queue default
2015-05-19 19:52:52,930 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from SUBMITTED to SCHEDULED
2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000001 Container Transitioned from NEW to ALLOCATED
2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=AM Allocated Container        TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1432064564266_0003       CONTAINERID=container_1432064564266_0003_01_000001
2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1432064564266_0003_01_000001 of capacity <memory:1024, vCores:1> on host ip-10-10-128-10.ec2.internal:9032, which has 1 containers, <memory:1024, vCores:1> used and <memory:7168, vCores:7> available after allocation
2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1432064564266_0003_000001 container=Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 clusterResource=<memory:8192, vCores:8>
2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.125, absoluteUsedCapacity=0.125, numApps=1, numContainers=1
2015-05-19 19:52:53,718 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.125 absoluteUsedCapacity=0.125 used=<memory:1024, vCores:1> cluster=<memory:8192, vCores:8>
2015-05-19 19:52:53,719 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : ip-10-10-128-10.ec2.internal:9032 for container : container_1432064564266_0003_01_000001
2015-05-19 19:52:53,720 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000001 Container Transitioned from ALLOCATED to ACQUIRED
2015-05-19 19:52:53,720 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Clear node set for appattempt_1432064564266_0003_000001
2015-05-19 19:52:53,721 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1432064564266_0003 AttemptId: appattempt_1432064564266_0003_000001 MasterContainer: Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.10.128.10:9032 }, ]
2015-05-19 19:52:53,721 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from SCHEDULED to ALLOCATED_SAVING
2015-05-19 19:52:53,721 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from ALLOCATED_SAVING to ALLOCATED
2015-05-19 19:52:53,723 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1432064564266_0003_000001
2015-05-19 19:52:53,725 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.10.128.10:9032 }, ] for AM appattempt_1432064564266_0003_000001
2015-05-19 19:52:53,725 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1432064564266_0003_01_000001 : {{JAVA_HOME}}/bin/java,-server,-Xmx512m,-Djava.io.tmpdir={{PWD}}/tmp,'-Dspark.fileserver.uri=http://10.10.127.10:8003','-Dspark.broadcast.port=8004','-Dspark.executor.port=8002','-Dspark.driver.appUIAddress=http://ip-10-10-127-10.ec2.internal:4040','-Dspark.fileserver.port=8003','-Dspark.ui.port=4040','-Dspark.tachyonStore.folderName=spark-ea0a49a2-1643-4410-892d-690c62cb6857','-Dspark.driver.extraJavaOptions=-Djava.net.preferIPv4Stack=true','-Dspark.jars=file:/home/testuser/spark/lib/spark-examples-1.3.1-hadoop2.6.0.jar','-Dspark.app.name=Spark Pi','-Dspark.akka.threads=1','-Dspark.authenticate.secret=fubar','-Dspark.eventlog.enabled=true','-Dspark.driver.host=ip-10-10-127-10.ec2.internal','-Dspark.blockManager.port=8007','-Dspark.history.ui.port=18080','-Dspark.replClassServer.port=8006','-Dspark.worker.ui.port=8081','-Dspark.master=yarn-client','-Dspark.yarn.preserve.staging.files=true','-Dspark.executor.extraJavaOptions=-XX:+PrintGCDetails -Dkey=value -Dnumbers=\"one two three\"','-Dspark.yarn.access.namenodes=hdfs://10.10.10.10:8020','-Dspark.logConf=true','-Dspark.driver.port=8001','-Dspark.executor.id=<driver>','-Dspark.akka.heartbeat.interval=100','-Dspark.master.ui.port=8080','-Dspark.yarn.am.waitTime=200000','-Dspark.yarn.am.extraJavaOptions=-Djava.net.preferIPv4Stack=true','-Dspark.executor.instances=1','-Dspark.yarn.am.port=8008','-Dspark.eventlog.dir=hdfs://10.10.10.10:8020/user/testuser/spark/eventlog','-Dspark.executor.cores=1','-Dspark.authenticate=true','-Djava.net.preferIPv4Stack=true',-Dspark.yarn.app.container.log.dir=<LOG_DIR>,org.apache.spark.deploy.yarn.ExecutorLauncher,--arg,'ip-10-10-127-10.ec2.internal:8001',--executor-memory,1024m,--executor-cores,1,--num-executors ,1,1>,<LOG_DIR>/stdout,2>,<LOG_DIR>/stderr
2015-05-19 19:52:53,725 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1432064564266_0003_000001
2015-05-19 19:52:53,725 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1432064564266_0003_000001
2015-05-19 19:52:53,750 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.10.128.10:9032 }, ] for AM appattempt_1432064564266_0003_000001
2015-05-19 19:52:53,750 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from ALLOCATED to LAUNCHED
2015-05-19 19:52:54,733 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000001 Container Transitioned from ACQUIRED to RUNNING
2015-05-19 19:53:17,318 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1432064564266_0003_000001 (auth:SIMPLE)
2015-05-19 19:53:17,376 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user appattempt_1432064564266_0003_000001: id: appattempt_1432064564266_0003_000001: no such user

2015-05-19 19:53:17,376 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user appattempt_1432064564266_0003_000001
2015-05-19 19:53:17,376 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for appattempt_1432064564266_0003_000001 (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB
2015-05-19 19:53:17,377 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AM registration appattempt_1432064564266_0003_000001
2015-05-19 19:53:17,377 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser IP=10.10.128.10 OPERATION=Register App Master   TARGET=ApplicationMasterService RESULT=SUCCESS     APPID=application_1432064564266_0003    APPATTEMPTID=appattempt_1432064564266_0003_000001
2015-05-19 19:53:17,377 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Setting client token master key
2015-05-19 19:53:17,378 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from LAUNCHED to RUNNING
2015-05-19 19:53:17,378 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from ACCEPTED to RUNNING
2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000002 Container Transitioned from NEW to ALLOCATED
2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=AM Allocated Container        TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1432064564266_0003       CONTAINERID=container_1432064564266_0003_01_000002
2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1432064564266_0003_01_000002 of capacity <memory:2048, vCores:1> on host ip-10-10-128-10.ec2.internal:9032, which has 2 containers, <memory:3072, vCores:2> used and <memory:5120, vCores:6> available after allocation
2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1432064564266_0003_000001 container=Container: [ContainerId: container_1432064564266_0003_01_000002, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:2048, vCores:1>, Priority: 1, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.125, absoluteUsedCapacity=0.125, numApps=1, numContainers=1 clusterResource=<memory:8192, vCores:8>
2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:3072, vCores:2>, usedCapacity=0.375, absoluteUsedCapacity=0.375, numApps=1, numContainers=2
2015-05-19 19:53:18,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.375 absoluteUsedCapacity=0.375 used=<memory:3072, vCores:2> cluster=<memory:8192, vCores:8>
2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1432064564266_0003_000001 with final state: FINISHING, and exit status: -1000
2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from RUNNING to FINAL_SAVING
2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1432064564266_0003 with final state: FINISHING
2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from RUNNING to FINAL_SAVING
2015-05-19 19:53:18,967 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1432064564266_0003
2015-05-19 19:53:18,968 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from FINAL_SAVING to FINISHING
2015-05-19 19:53:18,968 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from FINAL_SAVING to FINISHING
2015-05-19 19:53:19,076 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: application_1432064564266_0003 unregistered successfully.
2015-05-19 19:53:20,383 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000001 Container Transitioned from RUNNING to COMPLETED
2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1432064564266_0003_01_000001 in state: COMPLETED event:FINISHED
2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=AM Released Container TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1432064564266_0003       CONTAINERID=container_1432064564266_0003_01_000001
2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1432064564266_0003_01_000001 of capacity <memory:1024, vCores:1> on host ip-10-10-128-10.ec2.internal:9032, which currently has 1 containers, <memory:2048, vCores:1> used and <memory:6144, vCores:7> available, release resources=true
2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:2048, vCores:1> numContainers=1 user=testuser user-resources=<memory:2048, vCores:1>
2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1432064564266_0003_01_000001, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.10.128.10:9032 }, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:2048, vCores:1>, usedCapacity=0.25, absoluteUsedCapacity=0.25, numApps=1, numContainers=1 cluster=<memory:8192, vCores:8>
2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.25 absoluteUsedCapacity=0.25 used=<memory:2048, vCores:1> cluster=<memory:8192, vCores:8>
2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:2048, vCores:1>, usedCapacity=0.25, absoluteUsedCapacity=0.25, numApps=1, numContainers=1
2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1432064564266_0003_000001 released container container_1432064564266_0003_01_000001 on node: host: ip-10-10-128-10.ec2.internal:9032 #containers=1 available=6144 used=2048 with event: FINISHED
2015-05-19 19:53:20,384 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1432064564266_0003_000001
2015-05-19 19:53:20,385 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1432064564266_0003_000001
2015-05-19 19:53:20,385 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1432064564266_0003_000001 State change from FINISHING to FINISHED
2015-05-19 19:53:20,385 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1432064564266_0003 State change from FINISHING to FINISHED
2015-05-19 19:53:20,385 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1432064564266_0003_000001 is done. finalState=FINISHED
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1432064564266_0003_01_000002 Container Transitioned from ALLOCATED to KILLED
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1432064564266_0003_01_000002 in state: KILLED event:KILL
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=AM Released Container TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1432064564266_0003       CONTAINERID=container_1432064564266_0003_01_000002
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1432064564266_0003_01_000002 of capacity <memory:2048, vCores:1> on host ip-10-10-128-10.ec2.internal:9032, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:0, vCores:0> numContainers=0 user=testuser user-resources=<memory:0, vCores:0>
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1432064564266_0003_01_000002, NodeId: ip-10-10-128-10.ec2.internal:9032, NodeHttpAddress: ip-10-10-128-10.ec2.internal:8090, Resource: <memory:2048, vCores:1>, Priority: 1, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 cluster=<memory:8192, vCores:8>
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:8192, vCores:8>
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1432064564266_0003_000001 released container container_1432064564266_0003_01_000002 on node: host: ip-10-10-128-10.ec2.internal:9032 #containers=0 available=8192 used=0 with event: KILL
2015-05-19 19:53:20,387 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1432064564266_0003 requests cleared
2015-05-19 19:53:20,388 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1432064564266_0003 user: testuser queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2015-05-19 19:53:20,388 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application removed - appId: application_1432064564266_0003 user: testuser leaf-queue of parent: root #applications: 0
2015-05-19 19:53:20,388 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=testuser OPERATION=Application Finished - Succeeded      TARGET=RMAppManager     RESULT=SUCCESS     APPID=application_1432064564266_0003
2015-05-19 19:53:20,389 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1432064564266_0003,name=Spark Pi,user=testuser,queue=default,state=FINISHED,trackingUrl=https://ip-10-10-127-10.ec2.internal:8090/proxy/application_1432064564266_0003/A,appMasterHost=ip-10-10-128-10.ec2.internal,startTime=1432065172758,finishTime=1432065198967,finalStatus=SUCCEEDED
2015-05-19 19:53:20,389 INFO org.apache.hadoop.hdfs.DFSClient: Cancelling HDFS_DELEGATION_TOKEN token 5 for testuser on 10.10.10.10:8020
2015-05-19 19:53:20,390 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1432064564266_0003_000001
2015-05-19 19:53:20,442 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Null container completed...
root@ip-10-10-127-10:/var/log/hadoop>

Keith Nance
,

How do I integrate Hadoop app development with Eclipse IDE?

1. create pom.xml for your project
2. add hadoop dependencies which you need
3. $ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
4. import existing java project to eclipse


On Wed, May 20, 2015 at 5:31 PM, Caesar Samsi wrote:
Hello,

I'm embarking on my first tutorial and would like to have tooltip help as I hover my mouse pointer over Hadoop classes.

I've found the Hadoop docs and Javadoc URL and configured them but the tooltips still don't show up.

Thanks you, Caesar.

Wednesday, May 20, 2015

, , ,

“dfs.namenode.service.handler.count” and “dfs.namenode.handler.count”

Hi jason,

Looked around the source-code, both parameters are not deprecated.

dfs.namenode.service.handler.count : Specifies the number of
threads for NameNode to handle RPC requests from DataNodes,
standby NameNode, and all other non-client nodes
(BackupNode and SecondaryNameNode). Default to 10.
This parameter is valid only if dfs.namenode.servicerpc-address
is configured. If the parameter is configured, NameNode
initializes extra RPC server to handle requests from
non-client nodes.

dfs.namenode.handler.count : Specifies the number of threads
for NameNode to handle RPC requests. Default to 10.

I'll file a jira to document dfs.namenode.service.handler.count.

Regards,
Akira

On 05/20/2015 12:30 PM, jason lu wrote:
Hi,

 what is the difference between "dfs.namenode.service.handler.count" and "dfs.namenode.handler.count" in hdfs-site.xml?
 I found this in stackoverflow:

  • The RPC Server needs threads to handle requests- Hadoop ships with his own RPC framework and you can configure this with the dfs.namenode.service.handler.count for the datanodes it is default set to 10. Or you can configure this dfs.namenode.handler.count for other clients, like MapReduce jobs, JobClients that want to run a job. When a request comes in and it want to create a new handler, it go may out of memory (new Threads are also allocating a good chunk of stack space, maybe you need to increase this).


  • I can't  find the dfs.namenode.service.handler.count in hdfs-site.xml of hadoop 2.3, is is deprecated? but I found it in the source.

    BTW, any suggestion for these parameters?

    thanks.



    , ,

    how namenode decided which datanode are ready for a client

    but on what category datanode are choosen by namenode, if some client want to write some data(block) on it via HDFS
    how namenode distinguish on one datanode and the other if only I just have one rack, so rack awareness feature are unused

    On Tue, May 19, 2015 at 3:56 PM, rajesh damera wrote:
    Namenode usually checks for the jobs running on the datanodes, of which tasktrackers keep a track of and decides if it still busy running a task.
    On May 19, 2015 14:23, "rajesh damera" wrote:
    Datanode availability is monitored and decided by Namenode. You usually get permissions, based on the type of user you are, to use a certain number of nodes to run your jobs. These limitations are set by administrators at user level
    Warm regards
    Rajesh Damera
    On May 19, 2015 14:08, "Himawan Mahardianto" wrote:
    Hi guys, I know that every datanode where sending heartbeat to prove that their alive but what method are used by namenode to decide which datanode are available to client if all datanode where available?

    Best Regards

    , , , ,

    Web Address appears to be ignored

    Thanks for the feedback Ravi. I entered a JIRA:

    https://issues.apache.org/jira/browse/HADOOP-11999

    -Ewan

    On 05/19/2015 07:45 PM, Ravi Prakash wrote:
    Ewan!

    This sounds like a bug. Please open a JIRA.

    Thanks
    Ravi



    On Tuesday, May 19, 2015 8:09 AM, Ewan Higgs wrote:


    Hi all,
    I am setting up a Hadoop cluster where the nodes have FQDNames inside
    the cluster, but the DNS where these names are registered is behind some
    login nodes. So any user who tries to access the web interface needs to
    use the IPs instead.

    I set the 'yarn.nodemanager.webapp.address' and
    'yarn.resourcemanager.webapp.address' to the appropriate IP:port. I
    don't give it the FQDN in this config field.

    However, when I access the web app it all works inside each web app.
    However, when I cross from the Resource Manager to the Node Manager web
    app, the href url uses the FQDN that I don't want. Obviously this is a
    dead link to the user and can only be fixed if they copy and paste the
    appropriate IP address for the node (not a pleasant user experience).

    Is there a way to convince the web app to not use the FQDN or is this a
    potential bug? Or maybe this will end up as "WONTFIX - open up your DNS".

    Yours,
    Ewan



    Monday, May 18, 2015

    , , ,

    hadoop.tmp.dir?

    Hello,

    The 3 main settings in hdfs-site.xml are:

    •     dfs.name.dir: directory where namenode stores its metadata, default value ${hadoop.tmp.dir}/dfs/name.
    •     dfs.data.dir: directory where HDFS data blocks are stored, default value ${hadoop.tmp.dir}/dfs/data.
    •     dfs.namenode.checkpoint.dir: directory where secondary namenode store its checkpoints, default value is ${hadoop.tmp.dir}/dfs/namesecondary.


    By default it uses the ${hadoop.tmp.dir}:
    https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

    but one can provide comma-delimited list of directories paths to point to multiple locations/disks to have them distributed.

    HTH

    -Rajesh



    On Mon, May 18, 2015 at 2:41 PM, Caesar Samsi  wrote:
    Hello,

    Hadoop.tmp.dir seems to be the root of all storage directories.

    I'd like for data to be stored in separate locations.

    Is there a list of directories and how they can be specified?

    Thank you, Caesar.

    (.tmp seems to indicate a temporary condition and yet it's used by HDFS, etc.)

    Popular Posts