终于成功运行第一个Hadoop程序

 

之前一直只在Windows上玩 为了Hadoop 只好迁移到Ubuntu上～

 

 

记录一下自己的历程：

 

1. 在Windows上安装Ubuntu

 

方法就是通过wubi安装，全自动化，非常傻瓜，太适合我这种linux小白了！

 

wubi可以直接在官网下载，当时我不知道，使用的还是从iso之中解压出来的程序 哈哈

 

下载地址：http://www.ubuntu.com/download/desktop/windows-installer

 

自己选择合适的来用啊

 

 

 

 

2. 进行各种配置，包括： JDK Hadoop, 当然也包括安装一个拼音输入法。。。。

 

JDK以及配置Hadoop的方法，完全参考的：http://os.51cto.com/art/201211/364167.htm

 Note: 跟着上面的步骤走下来我才知道，这个实际上是伪分布式～

 

3. 开始跑一个例子程序吧～

hadoop自带的例子程序 就在解压出来的hadoop文件夹之中，但是一开始一直遇到一个很奇怪的问题：报错如下：

 

13/07/06 20:10:59 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://localhost:9000/usr/local/hadoop/input

org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://localhost:9000/usr/local/hadoop/input

at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235)

at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)

at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)

at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)

at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)

at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)

at org.apache.hadoop.examples.WordCount.main(WordCount.java:82)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)

at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)

at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

 

在各种google之后才知道，hdfs里面还没有把input这个文件夹加上去

 

敲入命令：

 

hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop dfs -put input input

 

这样之后，通过dfs的-ls命令终于能找到input文件夹：

 

hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop dfs -ls

Found 1 items

drwxr-xr-x - hadoop supergroup 0 2013-07-06 20:12 /user/hadoop/input

 

 

 

这个时候再次执行运行例子程序的命令，就没有之前遇到的异常：

 

Input path does not exist: hdfs://localhost:9000/usr/local/hadoop/input

 

最后正确执行的log如下：

 

hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop jar hadoop-examples-1.2.0.jar wordcount input wc_output2

13/07/06 20:12:39 INFO input.FileInputFormat: Total input paths to process : 1

13/07/06 20:12:39 INFO util.NativeCodeLoader: Loaded the native-hadoop library

13/07/06 20:12:39 WARN snappy.LoadSnappy: Snappy native library not loaded

13/07/06 20:12:40 INFO mapred.JobClient: Running job: job_201307061937_0006

13/07/06 20:12:41 INFO mapred.JobClient: map 0% reduce 0%

13/07/06 20:12:46 INFO mapred.JobClient: map 100% reduce 0%

13/07/06 20:12:53 INFO mapred.JobClient: map 100% reduce 33%

13/07/06 20:12:55 INFO mapred.JobClient: map 100% reduce 100%

13/07/06 20:12:55 INFO mapred.JobClient: Job complete: job_201307061937_0006

13/07/06 20:12:56 INFO mapred.JobClient: Counters: 29

13/07/06 20:12:56 INFO mapred.JobClient: Job Counters

13/07/06 20:12:56 INFO mapred.JobClient: Launched reduce tasks=1

13/07/06 20:12:56 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=4785

13/07/06 20:12:56 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

13/07/06 20:12:56 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0

13/07/06 20:12:56 INFO mapred.JobClient: Launched map tasks=1

13/07/06 20:12:56 INFO mapred.JobClient: Data-local map tasks=1

13/07/06 20:12:56 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=8917

13/07/06 20:12:56 INFO mapred.JobClient: File Output Format Counters

13/07/06 20:12:56 INFO mapred.JobClient: Bytes Written=24591

13/07/06 20:12:56 INFO mapred.JobClient: FileSystemCounters

13/07/06 20:12:56 INFO mapred.JobClient: FILE_BYTES_READ=34471

13/07/06 20:12:56 INFO mapred.JobClient: HDFS_BYTES_READ=46619

13/07/06 20:12:56 INFO mapred.JobClient: FILE_BYTES_WRITTEN=180119

13/07/06 20:12:56 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=24591

13/07/06 20:12:56 INFO mapred.JobClient: File Input Format Counters

13/07/06 20:12:56 INFO mapred.JobClient: Bytes Read=46510

13/07/06 20:12:56 INFO mapred.JobClient: Map-Reduce Framework

13/07/06 20:12:56 INFO mapred.JobClient: Map output materialized bytes=34471

13/07/06 20:12:56 INFO mapred.JobClient: Map input records=561

13/07/06 20:12:56 INFO mapred.JobClient: Reduce shuffle bytes=34471

13/07/06 20:12:56 INFO mapred.JobClient: Spilled Records=4992

13/07/06 20:12:56 INFO mapred.JobClient: Map output bytes=77170

13/07/06 20:12:56 INFO mapred.JobClient: Total committed heap usage (bytes)=219152384

13/07/06 20:12:56 INFO mapred.JobClient: CPU time spent (ms)=2450

13/07/06 20:12:56 INFO mapred.JobClient: Combine input records=7804

13/07/06 20:12:56 INFO mapred.JobClient: SPLIT_RAW_BYTES=109

13/07/06 20:12:56 INFO mapred.JobClient: Reduce input records=2496

13/07/06 20:12:56 INFO mapred.JobClient: Reduce input groups=2496

13/07/06 20:12:56 INFO mapred.JobClient: Combine output records=2496

13/07/06 20:12:56 INFO mapred.JobClient: Physical memory (bytes) snapshot=280190976

13/07/06 20:12:56 INFO mapred.JobClient: Reduce output records=2496

13/07/06 20:12:56 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2009939968

13/07/06 20:12:56 INFO mapred.JobClient: Map output records=7804

 

 

 

 

 

最后，查看运行完成之后的结果：

 

bin/hadoop dfs -cat wc_output2/* | more

 

 

 

备注：

 

几个GUI管理网址：［目前我知道的］

 

http://localhost:50070/dfshealth.jsp 

 

http://localhost:50030/jobtracker.jsp