词频统计之自定义Driver类实现
来源：4-7 词频统计之自定义Driver类实现
qq_奋斗_nDHX48
2020-02-23
[WARN ] method:org.apache.hadoop.util.NativeCodeLoader.(NativeCodeLoader.java:62)
Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
[ERROR] method:org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:425)
Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:407)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:422)
at org.apache.hadoop.util.Shell.(Shell.java:415)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:79)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:168)
at org.apache.hadoop.security.Groups.(Groups.java:132)
at org.apache.hadoop.security.Groups.(Groups.java:100)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:435)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:341)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:308)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:895)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:861)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:728)
at org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:72)
at org.apache.hadoop.mapreduce.Job.(Job.java:145)
at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:188)
at com.wts.bigdata.hadoop.mr.wc.WordCountApp.main(WordCountApp.java:24)
[INFO ] method:org.apache.hadoop.conf.Configuration.warnOnceIfDeprecated(Configuration.java:1274)
session.id is deprecated. Instead, use dfs.metrics.session-id
[INFO ] method:org.apache.hadoop.metrics.jvm.JvmMetrics.init(JvmMetrics.java:76)
Initializing JVM Metrics with processName=JobTracker, sessionId=
Exception in thread “main” java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:548)
at org.apache.hadoop.util.Shell.run(Shell.java:507)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:882)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:865)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:720)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:478)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:518)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:316)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:148)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)
at com.wts.bigdata.hadoop.mr.wc.WordCountApp.main(WordCountApp.java:46)
Process finished with exit code 1
idea显示以上错误
查看qq群和问答群给的答案不太理解 ，能详细说说为什么会报出这个错误，在win中运行也需要在win中下载并且配置hadoop吗？
2回答
-
qq_奋斗_nDHX48
提问者
2020-02-25
设置完后还是报出异常错误
[INFO ] method:org.apache.hadoop.conf.Configuration.warnOnceIfDeprecated(Configuration.java:1274)
session.id is deprecated. Instead, use dfs.metrics.session-id
[INFO ] method:org.apache.hadoop.metrics.jvm.JvmMetrics.init(JvmMetrics.java:76)
Initializing JVM Metrics with processName=JobTracker, sessionId=
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:465)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:518)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:316)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:148)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)
at com.wts.bigdata.hadoop.mr.wc.WordCountApp.main(WordCountApp.java:49)Process finished with exit code 1
NativeIO类重新编译了也，但是还是有错误。
112020-05-06 -
Michael_PK
2020-02-23
这个就是win机器跑mr作业的时候势必会遇到的一个问题，需要几个工具下载到win本地，然后配置到环境变量中就可以。具体的解决方案可以加到qq群里去，qq群文件中有解决方案。 当然这个错误也是可以回避的，你开发好，把代码打包到服务器上去执行就不会有这个问题的，或者你在linux中安装一个idea，然后开发，也不会有这问题存在。
00
相似问题