大數據集群間數據由hbase到impala

2024-07-21 02:51:47

字體：大中小

來源：轉載

供稿：網友

10.0.0.11創建測試表create 'LV_TEST',{NAME =>'cf' ,BLOOMFILTER => 'ROW',COMPRESSION => 'SNAPPY',VERSIONS => 1}插入測試數據put 'LV_TEST','rowkey001','cf:name','zhangsan'put 'LV_TEST','rowkey001','cf:age','30'get 'LV_TEST','rowkey001', {COLUMN=>'cf:name'}get 'LV_TEST','rowkey001'put 'LV_TEST','rowkey002','cf:name','lisi'put 'LV_TEST','rowkey002','cf:age','40'get 'LV_TEST','rowkey002'查看測試表scan 'LV_TEST'導出到HDFSfind ./ /opt -name hbasecd /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/binhbase org.apache.hadoop.hbase.mapreduce.Export LV_TEST /data/LV_TESThadoop fs -ls /data/LV_TESThadoop fs -get /data/LV_TEST然后通過FTP下載到本地上傳到另一服務器192.168.51.251上傳到HDFSsu - hdfshadoop fs -mkdir /datahadoop fs -put /LV_TEST /data/LV_TESThadoop fs -ls /data/LV_TEST創建Hbase表create 'LV_TEST_COPY',{NAME =>'cf' ,BLOOMFILTER => 'ROW',COMPRESSION => 'SNAPPY',VERSIONS => 1}scan 'LV_TEST_COPY'hbase org.apache.hadoop.hbase.mapreduce.Import LV_TEST_COPY /data/LV_TEST建立hive外表hiveCREATE EXTERNAL TABLE LV_TEST_COPY(id string,name string,age string)ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key, cf:name, cf:age")TBLPROPERTIES("hbase.table.name" = "LV_TEST_COPY");impalaimpala-shell -i node90同步元數據INVALIDATE METADATA;SQL查詢結果

select * from LV_TEST_COPY;

最后將外部表數據復制到基于parques的impala內部表，完成數據轉換。

insert into ***() select * from *****

上一篇：利用 ELK 處理 Percona 審計日志

下一篇：Redis的主從復制