11G数据的hive初测试

首先，看到标题，我想就会有人喷我，①11G对于hadoop而言不值一提②做耗时测试本没有必要，毕竟hadoop用于线下系统。

我的想法：没那么复杂，只是想看下hadoop的文件处理能力和表现出的问题，只是单纯的享受这个解决问题并且得到结果的快感。

不那么多废话啦!直接进入正题：

1、数据来源：

apache的日志，接近900万条数据，大小为11个G。

2、创建表，并用正则匹配的方式导入数据，(这里hive的正则和普通正则，路由不同，之后会发新的博客讨论这个问题：(hive的正则)http://blog.csdn.net/lengzijian/article/details/7048575)：

这里正则值提取的日志当中的：site：当前站点 ，get：当前站点请求页，fore：之前站点------主要用于点击流，淡然是简单版的。为了以后应付更大文件做准备。

CREATE TABLE click (site STRING, get STRING, fore STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ("input.regex" = ".*?- ([^ ]*) [GET]*[POST]* (/.*?) .*WEBTRENDS_ID=.* [http://]*(.+) .*","output.format.string" = "%1$s,%3$s,%2$s")STORED AS TEXTFILE;

3、导入数据，我这里有23个数据，分别代表一天的时间(可能少了一个小时)，可以把这23个数据连接在一起导入，但是我没有这样做，因为分别看每一个的时间，很容易的看到每个时间段的访问量，对于绘制监控图比较方便。

LOAD DATA LOCAL INPATH "/data/log/sdc_20110926_hr【00-22】.log" INTO TABLE click;

4、查询：因为刚刚做完没有经验，直接运行了

select count(*) from click;

出现以下信息：

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201112071106_0005, Tracking URL = http://hadoop149:50030/jobdetails.jsp?jobid=job_201112071106_0005
Kill Command = /root/hadoop-0.20.2/bin/hadoop job  -Dmapred.job.tracker=hdfs://192.168.30.149:9004 -kill job_201112071106_0005
2011-12-07 17:03:55,880 Stage-1 map = 0%,  reduce = 0%
2011-12-07 17:04:04,936 Stage-1 map = 1%,  reduce = 0%
2011-12-07 17:04:15,040 Stage-1 map = 2%,  reduce = 0%
2011-12-07 17:04:17,057 Stage-1 map = 3%,  reduce = 0%
2011-12-07 17:04:24,103 Stage-1 map = 4%,  reduce = 0%
2011-12-07 17:04:26,121 Stage-1 map = 5%,  reduce = 0%
2011-12-07 17:04:36,185 Stage-1 map = 6%,  reduce = 0%
2011-12-07 17:04:38,199 Stage-1 map = 7%,  reduce = 0%
2011-12-07 17:04:44,239 Stage-1 map = 7%,  reduce = 2%
2011-12-07 17:04:50,275 Stage-1 map = 9%,  reduce = 2%
2011-12-07 17:04:59,333 Stage-1 map = 11%,  reduce = 3%
2011-12-07 17:05:08,386 Stage-1 map = 13%,  reduce = 3%
2011-12-07 17:05:14,425 Stage-1 map = 13%,  reduce = 4%
2011-12-07 17:05:17,450 Stage-1 map = 14%,  reduce = 4%
2011-12-07 17:05:21,476 Stage-1 map = 15%,  reduce = 4%
2011-12-07 17:05:25,505 Stage-1 map = 16%,  reduce = 4%
2011-12-07 17:05:28,530 Stage-1 map = 16%,  reduce = 5%
2011-12-07 17:05:34,568 Stage-1 map = 17%,  reduce = 5%
2011-12-07 17:05:37,588 Stage-1 map = 18%,  reduce = 5%
2011-12-07 17:05:43,629 Stage-1 map = 19%,  reduce = 6%
2011-12-07 17:05:46,648 Stage-1 map = 20%,  reduce = 6%
2011-12-07 17:05:55,705 Stage-1 map = 22%,  reduce = 6%
2011-12-07 17:05:58,724 Stage-1 map = 22%,  reduce = 7%
2011-12-07 17:06:04,761 Stage-1 map = 23%,  reduce = 7%
2011-12-07 17:06:07,783 Stage-1 map = 24%,  reduce = 7%
2011-12-07 17:06:16,839 Stage-1 map = 25%,  reduce = 8%
2011-12-07 17:06:18,853 Stage-1 map = 26%,  reduce = 8%
2011-12-07 17:06:25,900 Stage-1 map = 27%,  reduce = 8%
2011-12-07 17:06:31,937 Stage-1 map = 27%,  reduce = 9%
2011-12-07 17:06:33,951 Stage-1 map = 28%,  reduce = 9%
2011-12-07 17:06:36,973 Stage-1 map = 29%,  reduce = 9%
2011-12-07 17:06:43,010 Stage-1 map = 30%,  reduce = 9%
2011-12-07 17:06:47,039 Stage-1 map = 31%,  reduce = 10%
2011-12-07 17:06:55,088 Stage-1 map = 32%,  reduce = 10%
2011-12-07 17:06:56,095 Stage-1 map = 33%,  reduce = 10%
2011-12-07 17:07:03,140 Stage-1 map = 34%,  reduce = 10%
2011-12-07 17:07:08,172 Stage-1 map = 35%,  reduce = 11%
2011-12-07 17:07:14,210 Stage-1 map = 36%,  reduce = 11%
2011-12-07 17:07:18,241 Stage-1 map = 37%,  reduce = 11%
2011-12-07 17:07:23,275 Stage-1 map = 38%,  reduce = 12%
2011-12-07 17:07:29,312 Stage-1 map = 39%,  reduce = 12%
2011-12-07 17:07:32,331 Stage-1 map = 40%,  reduce = 13%
2011-12-07 17:07:38,373 Stage-1 map = 41%,  reduce = 13%
2011-12-07 17:07:41,392 Stage-1 map = 42%,  reduce = 13%
2011-12-07 17:07:47,429 Stage-1 map = 42%,  reduce = 14%
2011-12-07 17:07:48,436 Stage-1 map = 43%,  reduce = 14%
2011-12-07 17:07:51,456 Stage-1 map = 44%,  reduce = 14%
2011-12-07 17:07:56,497 Stage-1 map = 45%,  reduce = 14%
2011-12-07 17:08:02,533 Stage-1 map = 46%,  reduce = 14%
2011-12-07 17:08:08,568 Stage-1 map = 46%,  reduce = 15%
2011-12-07 17:08:09,575 Stage-1 map = 47%,  reduce = 15%
2011-12-07 17:08:12,594 Stage-1 map = 48%,  reduce = 15%
2011-12-07 17:08:18,634 Stage-1 map = 49%,  reduce = 15%
2011-12-07 17:08:23,664 Stage-1 map = 50%,  reduce = 16%
2011-12-07 17:08:29,701 Stage-1 map = 51%,  reduce = 16%
2011-12-07 17:08:32,718 Stage-1 map = 51%,  reduce = 17%
2011-12-07 17:08:35,736 Stage-1 map = 52%,  reduce = 17%
2011-12-07 17:08:39,761 Stage-1 map = 53%,  reduce = 17%
2011-12-07 17:08:44,791 Stage-1 map = 54%,  reduce = 17%
2011-12-07 17:08:51,832 Stage-1 map = 55%,  reduce = 17%
2011-12-07 17:08:53,845 Stage-1 map = 55%,  reduce = 18%
2011-12-07 17:08:56,864 Stage-1 map = 56%,  reduce = 18%
2011-12-07 17:09:00,887 Stage-1 map = 57%,  reduce = 18%
2011-12-07 17:09:05,920 Stage-1 map = 58%,  reduce = 18%
2011-12-07 17:09:08,939 Stage-1 map = 58%,  reduce = 19%
2011-12-07 17:09:14,974 Stage-1 map = 60%,  reduce = 19%
2011-12-07 17:09:24,029 Stage-1 map = 61%,  reduce = 20%
2011-12-07 17:09:27,049 Stage-1 map = 62%,  reduce = 20%
2011-12-07 17:09:33,086 Stage-1 map = 63%,  reduce = 20%
2011-12-07 17:09:36,105 Stage-1 map = 64%,  reduce = 20%
2011-12-07 17:09:39,124 Stage-1 map = 64%,  reduce = 21%
2011-12-07 17:09:45,160 Stage-1 map = 65%,  reduce = 21%
2011-12-07 17:09:46,167 Stage-1 map = 66%,  reduce = 21%
2011-12-07 17:09:54,216 Stage-1 map = 67%,  reduce = 22%
2011-12-07 17:09:58,241 Stage-1 map = 68%,  reduce = 22%
2011-12-07 17:10:03,272 Stage-1 map = 69%,  reduce = 22%
2011-12-07 17:10:09,309 Stage-1 map = 69%,  reduce = 23%
2011-12-07 17:10:10,316 Stage-1 map = 70%,  reduce = 23%
2011-12-07 17:10:15,345 Stage-1 map = 71%,  reduce = 23%
2011-12-07 17:10:21,382 Stage-1 map = 72%,  reduce = 23%
2011-12-07 17:10:24,401 Stage-1 map = 73%,  reduce = 24%
2011-12-07 17:10:30,438 Stage-1 map = 74%,  reduce = 24%
2011-12-07 17:10:33,457 Stage-1 map = 75%,  reduce = 24%
2011-12-07 17:10:39,497 Stage-1 map = 76%,  reduce = 24%
2011-12-07 17:10:42,517 Stage-1 map = 77%,  reduce = 24%
2011-12-07 17:10:45,537 Stage-1 map = 77%,  reduce = 25%
2011-12-07 17:10:51,589 Stage-1 map = 79%,  reduce = 25%
2011-12-07 17:11:00,645 Stage-1 map = 80%,  reduce = 26%
2011-12-07 17:11:06,683 Stage-1 map = 81%,  reduce = 27%
2011-12-07 17:11:09,700 Stage-1 map = 82%,  reduce = 27%
2011-12-07 17:11:18,755 Stage-1 map = 83%,  reduce = 27%
2011-12-07 17:11:21,774 Stage-1 map = 84%,  reduce = 27%
2011-12-07 17:11:30,854 Stage-1 map = 85%,  reduce = 28%
2011-12-07 17:11:33,874 Stage-1 map = 86%,  reduce = 28%
2011-12-07 17:11:39,912 Stage-1 map = 87%,  reduce = 28%
2011-12-07 17:11:45,949 Stage-1 map = 88%,  reduce = 29%
2011-12-07 17:11:48,970 Stage-1 map = 89%,  reduce = 29%
2011-12-07 17:11:55,008 Stage-1 map = 90%,  reduce = 29%
2011-12-07 17:11:58,028 Stage-1 map = 91%,  reduce = 29%
2011-12-07 17:12:01,048 Stage-1 map = 91%,  reduce = 30%
2011-12-07 17:12:04,067 Stage-1 map = 92%,  reduce = 30%
2011-12-07 17:12:07,088 Stage-1 map = 93%,  reduce = 30%
2011-12-07 17:12:13,127 Stage-1 map = 94%,  reduce = 31%
2011-12-07 17:12:19,167 Stage-1 map = 95%,  reduce = 31%
2011-12-07 17:12:22,186 Stage-1 map = 96%,  reduce = 31%
2011-12-07 17:12:25,205 Stage-1 map = 97%,  reduce = 31%
2011-12-07 17:12:28,225 Stage-1 map = 98%,  reduce = 31%
2011-12-07 17:12:31,251 Stage-1 map = 99%,  reduce = 31%
2011-12-07 17:12:34,275 Stage-1 map = 100%,  reduce = 33%
2011-12-07 17:12:40,312 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201112071106_0005
OK
8625654
Time taken: 541.638 seconds

在执行过程中，一台电脑(子节点)一直再响，看了下cpu，双核几乎全被占满。最后结果运行了

<pre name="code" class="plain">OK
8625654
Time taken: 541.638 seconds</pre>

测试先做到这里，之后会继续更新。