先週、ユーザー「root」を使用して Hadoop の dfs と mapreduce を開始し、Embedded-Pig Java コードを実行しました。元気に走っていました。今週は、root 以外のユーザー charlieを使用して同じタスクを実行したいと思います。いくつかのディレクトリのユーザー権限設定を変更した後、ユーザー「charlie」を使用して Hadoop の dfs と mapreduce をエラーなしで開始できるようになりました。ただし、ユーザー「charlie」を使用して Embedded-Pig Java コードを実行すると、core-stie.xml で /opt/hdfs/tmp/ に設定した hadoop.tmp.dir の許可に文句を言い続けます。
java.io.FileNotFoundException: /opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (許可が拒否されました)
次のディレクトリのアクセス許可を確認しましたが、すべて問題ないようです。
bash-3.2$ ls -lt /opt/hdfs/tmp
total 4
drwxr-xr-x 3 charlie comusers 4096 Apr 16 19:30 mapred
bash-3.2$ ls -lt /opt/hdfs/tmp/mapred
total 4
drwxr-xr-x 2 charlie comusers 4096 Apr 16 19:30 local
bash-3.2$ ls -lt /opt/hdfs/tmp/mapred/local
total 0
私が間違っていることについて何らかの方向性が必要です。これらのキーワードをグーグルで検索しましたが、何も見つかりませんでした。どんな助けでも大歓迎です！
次のように豚の出力を添付しました。情報が役立つことを願っています。
12/04/16 19:31:28 INFO executionengine.HExecutionEngine: Connecting to hadoop file system at: hdfs://hadoop-namenode:9000
12/04/16 19:31:29 INFO pigstats.ScriptState: Pig features used in the script: HASH_JOIN,GROUP_BY,FILTER,CROSS
12/04/16 19:31:29 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/04/16 19:31:29 INFO mapReduceLayer.MRCompiler: File concatenation threshold: 100 optimistic? false
12/04/16 19:31:30 INFO mapReduceLayer.CombinerOptimizer: Choosing to move algebraic foreach to combiner
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size before optimization: 11
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 map-reduce splittees.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 2 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 2 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 1 map-reduce splittees.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 1 out of total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size after optimization: 10
12/04/16 19:31:30 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:30 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:30 INFO pigstats.ScriptState: Pig script settings are added to the job
12/04/16 19:31:30 WARN pigstats.ScriptState: unable to read pigs manifest file
12/04/16 19:31:30 INFO mapReduceLayer.JobControlCompiler: mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
12/04/16 19:31:35 INFO mapReduceLayer.JobControlCompiler: Setting up multi store job
12/04/16 19:31:35 INFO mapReduceLayer.JobControlCompiler: BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=957600
12/04/16 19:31:35 INFO mapReduceLayer.JobControlCompiler: Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce job(s) waiting for submission.
12/04/16 19:31:35 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO input.FileInputFormat: Total input paths to process : 1
12/04/16 19:31:35 INFO util.MapRedUtil: Total input paths to process : 1
12/04/16 19:31:35 INFO util.MapRedUtil: Total input paths (combined) to process : 1
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: 0% complete
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: job null has failed! Stop running all dependent jobs
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: 100% complete
12/04/16 19:31:36 WARN mapReduceLayer.Launcher: There is no log file to write to.
12/04/16 19:31:36 ERROR mapReduceLayer.Launcher: Backend error message during job submission
java.io.FileNotFoundException: /opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:180)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:176)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:234)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:368)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:465)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:372)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1216)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:92)
at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:373)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:662)
12/04/16 19:31:36 ERROR pigstats.SimplePigStats: ERROR 2997: Unable to recreate exception from backend error: java.io.FileNotFoundException: /opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission denied)
12/04/16 19:31:36 ERROR pigstats.PigStatsUtil: 1 map reduce job(s) failed!
12/04/16 19:31:36 WARN pigstats.ScriptState: unable to read pigs manifest file
12/04/16 19:31:36 INFO pigstats.SimplePigStats: Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2 charlie 2012-04-16 19:31:30 2012-04-16 19:31:36 HASH_JOIN,GROUP_BY,FILTER,CROSS
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
N/A events,events1,grouped MULTI_QUERY Message: java.io.FileNotFoundException: /opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:180)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:176)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:234)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:368)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:465)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:372)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1216)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:92)
at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:373)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:662)
Input(s):
Failed to read data from "/grapevine/analysis/recommendation/input/article_based/all_grapevine_events.txt"
Output(s):
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
null -> null,null,
null -> null,
null -> null,
null -> null,null,
null -> null,
null -> null,null,
null -> null,
null -> null,
null -> null,
null
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: Failed!