windows10_x64下配置hadoop2.7.5的环境搭建介绍

作者: admin 分类: Hadoop 发布时间: 2018-02-08 17:50  阅读: 541 views

windows10_x64下配置hadoop2.7.5环境工具等准备(一)

系统环境: window10 64位
java jdk : 1.8.0_121
Hadoop : 2.7.5 (注意环境变量的配置)
链接: https://pan.baidu.com/s/1jKh9Up4 密码: mbtg
依赖插件: windows上需要的winutils【Hadoop在运行中需要的一个服务进程】支持 和hadoop.dll【在windows环境下运行所需要的文件】等文件
链接: https://pan.baidu.com/s/1pMvwCGV 密码: f6rx
Eclipse :4.6.3版本
eclip插件 :hadoop-eclipse-plugin-2.7.1.jar 工具包
链接: https://pan.baidu.com/s/1i5VQ0mx 密码: wpd5
配置文件 : hadoop/etc/hadoop/下
core-site.xml/ hdfs-site.xml/ mapred-site.xml/ yarn-site.xml/ hadoop-env.cmd/sh
运行文件:诸多人的入门 WordCount.java
链接: https://pan.baidu.com/s/1qZ75QDA 密码: 7gzi
 
 

 

windows10_x64下配置hadoop2.7.5环境出现的问题(二)

 
A类:从官网下载好hadoop文件后,启动hadoop/sbin/start_all.cmd文件的包错问题
错误1:org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory C:\hadoop\workplace\tmp\dfs\name is in an inconsistent state: storage directory does not exist or is not accessible.
处理1:这个是和hadoop/etc/hadoop/文件夹中的 hdfs-site.xml 和 core-site.xml中的配置有关系。文件中的对应dir目录。实际磁盘中也要存在。对应建立好之后,问题解决。
错误2:Could not locate executable null \bin\winutils.exe in the hadoop binaries
处理2:缺少对应的winutils.exe程序【Hadoop在运行中需要的一个服务进程】
错误3:Unable to load native-hadoop library for your platform… using builtin-Java classes where applicable
处理3:缺少对应版本的hadoop.dll文件。放置于windows/system32/目录中,我的hadoop是2.7.5.dll文件找的是2.7版本64位的。 如果版本不对,系统位数不对,可能引起其他问题,我没遇到。
错误4:'MySQL' 不是内部或外部命令,也不是可运行的程序
处理4:这是一个很奇怪的问题,hadoop框架服务和mysql有什么关系。我本地安装了mysql5.6,开机时自动启动。后来把环境变量中的path属性删除了mysql相关,重启正常。【怀疑是加载环境变量时有冲突,没有细致研究】
错误5:ExitCodeException exitCode=-1073741515:
处理5:https://www.cnblogs.com/garfieldcgf/p/5482203.html,因为改了其他问题后,没有出现这个,所有没有记录了。
错误6:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
处理6:自行百度吧,这个我忽略他了,不影响正常运行。

 

B类:hadoop项目本身已经启动成功,但是用eclipse启动Map/Reduce项目失败
错误1:日志没有输出
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
处理1:日志要在控制台输出,需要在当前项目src目录下有日志配置[把hadoop/etc/hadoop/log4j.properties文件拷贝到项目src目录下即可]
错误2:Map/Reduce locations配置后【配置准确】,但 DFS locations 无法正常连接
处理2:要在HDFS文件系统中,创建对应的input文件夹。
错误3:Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:9000/user/xxx/output already exists
处理3:output文件夹已经生成了结果,需要删除才可以。

 

 

windows10环境下配置hadoop2.7.5运行成功的标志(三)

1.hadoop/sbin/start-all.cmd能够正常启动,无报错

本地hapoop搭建好之后能正常访问以下地址(非开发环境运行程序)

 (一)http://localhost:8088/ 资源管理GUI   (二)http://localhost:50070/ 节点管理GUI

2.Map/Reduce项目运行日志能正确出来
—————————————————————————————————————————————–
18/02/08 17:43:15 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
18/02/08 17:43:15 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
18/02/08 17:43:16 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/02/08 17:43:16 WARN mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
18/02/08 17:43:16 INFO input.FileInputFormat: Total input paths to process : 1
18/02/08 17:43:16 INFO mapreduce.JobSubmitter: number of splits:1
18/02/08 17:43:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1262378783_0001
18/02/08 17:43:16 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
18/02/08 17:43:16 INFO mapreduce.Job: Running job: job_local1262378783_0001
18/02/08 17:43:16 INFO mapred.LocalJobRunner: OutputCommitter set in config null
18/02/08 17:43:16 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
18/02/08 17:43:16 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
18/02/08 17:43:17 INFO mapred.LocalJobRunner: Waiting for map tasks
18/02/08 17:43:17 INFO mapred.LocalJobRunner: Starting task: attempt_local1262378783_0001_m_000000_0
18/02/08 17:43:17 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
18/02/08 17:43:17 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
18/02/08 17:43:17 INFO mapred.Task: Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@295ee683
18/02/08 17:43:17 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/deathearth/input/test.txt:0+1465
18/02/08 17:43:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
18/02/08 17:43:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
18/02/08 17:43:17 INFO mapred.MapTask: soft limit at 83886080
18/02/08 17:43:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
18/02/08 17:43:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
18/02/08 17:43:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
18/02/08 17:43:17 INFO mapred.LocalJobRunner:
18/02/08 17:43:17 INFO mapred.MapTask: Starting flush of map output
18/02/08 17:43:17 INFO mapred.MapTask: Spilling map output
18/02/08 17:43:17 INFO mapred.MapTask: bufstart = 0; bufend = 2544; bufvoid = 104857600
18/02/08 17:43:17 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213344(104853376); length = 1053/6553600
18/02/08 17:43:17 INFO mapred.MapTask: Finished spill 0
18/02/08 17:43:17 INFO mapred.Task: Task:attempt_local1262378783_0001_m_000000_0 is done. And is in the process of committing
18/02/08 17:43:17 INFO mapred.LocalJobRunner: map
18/02/08 17:43:17 INFO mapred.Task: Task 'attempt_local1262378783_0001_m_000000_0' done.
18/02/08 17:43:17 INFO mapred.Task: Final Counters for attempt_local1262378783_0001_m_000000_0: Counters: 22
File System Counters
FILE: Number of bytes read=179
FILE: Number of bytes written=302771
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1465
HDFS: Number of bytes written=0
HDFS: Number of read operations=5
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=5
Map output records=264
Map output bytes=2544
Map output materialized bytes=3078
Input split bytes=117
Combine input records=0
Spilled Records=264
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=7
Total committed heap usage (bytes)=271056896
File Input Format Counters
Bytes Read=1465
18/02/08 17:43:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local1262378783_0001_m_000000_0
18/02/08 17:43:17 INFO mapred.LocalJobRunner: map task executor complete.
18/02/08 17:43:17 INFO mapred.LocalJobRunner: Waiting for reduce tasks
18/02/08 17:43:17 INFO mapred.LocalJobRunner: Starting task: attempt_local1262378783_0001_r_000000_0
18/02/08 17:43:17 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
18/02/08 17:43:17 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
18/02/08 17:43:17 INFO mapred.Task: Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@a7e4ab4
18/02/08 17:43:17 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3fdbe65e
18/02/08 17:43:17 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=1319370752, maxSingleShuffleLimit=329842688, mergeThreshold=870784704, ioSortFactor=10, memToMemMergeOutputsThreshold=10
18/02/08 17:43:17 INFO reduce.EventFetcher: attempt_local1262378783_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
18/02/08 17:43:17 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1262378783_0001_m_000000_0 decomp: 3074 len: 3078 to MEMORY
18/02/08 17:43:17 INFO reduce.InMemoryMapOutput: Read 3074 bytes from map-output for attempt_local1262378783_0001_m_000000_0
18/02/08 17:43:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 3074, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->3074
18/02/08 17:43:17 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
18/02/08 17:43:17 INFO mapred.LocalJobRunner: 1 / 1 copied.
18/02/08 17:43:17 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
18/02/08 17:43:17 INFO mapred.Merger: Merging 1 sorted segments
18/02/08 17:43:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 3067 bytes
18/02/08 17:43:17 INFO reduce.MergeManagerImpl: Merged 1 segments, 3074 bytes to disk to satisfy reduce memory limit
18/02/08 17:43:17 INFO reduce.MergeManagerImpl: Merging 1 files, 3078 bytes from disk
18/02/08 17:43:17 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
18/02/08 17:43:17 INFO mapred.Merger: Merging 1 sorted segments
18/02/08 17:43:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 3067 bytes
18/02/08 17:43:17 INFO mapred.LocalJobRunner: 1 / 1 copied.
18/02/08 17:43:17 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
18/02/08 17:43:17 INFO mapreduce.Job: Job job_local1262378783_0001 running in uber mode : false
18/02/08 17:43:17 INFO mapreduce.Job: map 100% reduce 0%
18/02/08 17:43:18 INFO mapred.Task: Task:attempt_local1262378783_0001_r_000000_0 is done. And is in the process of committing
18/02/08 17:43:18 INFO mapred.LocalJobRunner: 1 / 1 copied.
18/02/08 17:43:18 INFO mapred.Task: Task attempt_local1262378783_0001_r_000000_0 is allowed to commit now
18/02/08 17:43:18 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1262378783_0001_r_000000_0' to hdfs://localhost:9000/user/deathearth/output/_temporary/0/task_local1262378783_0001_r_000000
18/02/08 17:43:18 INFO mapred.LocalJobRunner: reduce > reduce
18/02/08 17:43:18 INFO mapred.Task: Task 'attempt_local1262378783_0001_r_000000_0' done.
18/02/08 17:43:18 INFO mapred.Task: Final Counters for attempt_local1262378783_0001_r_000000_0: Counters: 29
File System Counters
FILE: Number of bytes read=6367
FILE: Number of bytes written=305849
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1465
HDFS: Number of bytes written=1335
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=156
Reduce shuffle bytes=3078
Reduce input records=264
Reduce output records=156
Spilled Records=264
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
Total committed heap usage (bytes)=271056896
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=1335
18/02/08 17:43:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local1262378783_0001_r_000000_0
18/02/08 17:43:18 INFO mapred.LocalJobRunner: reduce task executor complete.
18/02/08 17:43:18 INFO mapreduce.Job: map 100% reduce 100%
18/02/08 17:43:18 INFO mapreduce.Job: Job job_local1262378783_0001 completed successfully
18/02/08 17:43:18 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=6546
FILE: Number of bytes written=608620
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2930
HDFS: Number of bytes written=1335
HDFS: Number of read operations=13
HDFS: Number of large read operations=0
HDFS: Number of write operations=4
Map-Reduce Framework
Map input records=5
Map output records=264
Map output bytes=2544
Map output materialized bytes=3078
Input split bytes=117
Combine input records=0
Combine output records=0
Reduce input groups=156
Reduce shuffle bytes=3078
Reduce input records=264
Reduce output records=156
Spilled Records=528
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=7
Total committed heap usage (bytes)=542113792
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1465
File Output Format Counters
Bytes Written=1335
—————————————————————————————————————————————–
3.output文件中有最终的计算结果
—————————————————————————————————————————————–
say 1
score 1
sense, 1
seven 1
shall 3
should 1
so 3
struggled 1
take 1
task 1
testing 1
that 10
the 9
their 1
these 2
they 3

   原创文章,转载请标明本文链接: windows10_x64下配置hadoop2.7.5的环境搭建介绍

如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!

发表评论

电子邮件地址不会被公开。 必填项已用*标注

更多阅读