hdfs - Inserting record in Hive by appending -


i executing insert statements on hive external table. noticed each new insert, there new file created in hdfs path referenced external table. questions are:

  1. is possible have new inserted data append instead of creating new files?

  2. can control in such way size, 1mb, hive create new file place incoming inserts?

cloudera says:

the insert syntax appends data table. existing data files left as-is, , inserted data put 1 or more new data files.

hive append table not underlying files.

you can coerce hive build tables using create table, , forcing reducers one. copy fragmented files in 1 table , combine them in location in hdfs. swap files in hdfs.

you place files in holding area, check size of files there in hdfs , move them on above. seems easier temporarily hold files on local file system , move them over.

to combine files new file using hive can try:

set hive.exec.dynamic.partition.mode=nostrict; set hive.exec.compress.intermediate=false; set hive.exec.compress.output=false; set hive.exec.reducers.max=1;  create table if not exists db.table stored textfiel select * db.othertable;

db.othertable table has multiple fragemented files. db.table have single text file containing combined data.

i know not ideal , more of work around.


Comments

Post a Comment

Popular posts from this blog

toolbar - How to add link to user registration inside toobar in admin joomla 3 custom component -

im trying write component admin side , need add user creation. dont want duplicate existing codes add link. know how add task not simple link toolbar. thx help. if wanting button opens new user screen can try in component: $bar = jtoolbar::getinstance('toolbar'); $bar->appendbutton('link', 'users', 'new user','index.php?option=com_users&task=user.add'); you need add view.html.php file wherever you're creating toolbar. is need?
Read more

linux - disk space limitation when creating war file -

i have 1 gb disk space limitation hosting app , have used 700mb of it. however want create war file directory big (around 600mb) using command : jar -cvf root.war folderpath that's why encounter quota limitation , can't following. i'm looking way delete folder content when it's applying archive conserve disk space take note can't install zip tool in ssh shell ~ the code works on test directories, test first anyway, deletes files. it's not efficient. assumptions: 300mb free space, in destructively compress 600mb folderpath directory. none of files big -- if file in folderpath 300mb or larger, fail. notes: if ' $out ' file exists, nothing. first find pipes file list while , if ' $out ' doesn't exist in loop, create it, don't update -- if does exist, update it. each pass adds 1 more file, deletes unless file directory. after loop finishes delete empty directories too. unset u ; in=folderpath ; out=roo...
Read more