Deployment of artifacts to Hadoop cluster -


is there pattern how deploy applications (jar-files) hadoop-custer ? not talking map-reduce jobs deploy applications spark, flume etc.

within hadoop ecosystem deployment alone not sufficient. need restart services, deploy configurations (e.g. via ambari) , forth.

i have not found specific tools. assumption correct use standard automation tools maven/jenkins , missing parts ?

just wondering if have overseen something. not want reinvent wheel ;)

if managing hadoop ecosystem can use ambari , cloudera's manager. need stop , restart services configuration , library changes. if ecosystem managed outside of have option of managing jars outside tools puppet , salt. currently, use salt because of push/pull abilities.

if talking applications, jobs running on spark, provide hadoop url in file path. example: spark-submit --class my.dev.org.sparkdriver --properties-file mysparkprops.conf wordcount-shaded.jar hdfs://servername/input/file/sample.txt hdfs://servername/output/sparkresults

for applications have dependencies on third party jar files. have option of shading job's jar file prevent other application libraries interfering each other. down side application jar file big. use maven, added maven-shade-plugin artifact , use default scope (compile) dependencies.


Comments

Post a Comment

Popular posts from this blog

How to provide Authorization & Authentication using Asp.net, C#? -

Image
i have asp.net application in c#. i have provide page authorization specific user/usergroup. i have created 2 tables in database: 1) pagename 2)permissions[insert, update, view, delete] but i'm unable match tables , permissions assigning... can 1 please me, how created tables in database , how coding. please me. you can below explanation. configure access specific file , folder, set forms-based authentication. request page in application redirected logon.aspx automatically. in web.config file, type or paste following code. this code grants users access default1.aspx page , subdir1 folder . <configuration> <system.web> <authentication mode="forms" > <forms loginurl="login.aspx" name=".aspnetauth" protection="none" path="/" timeout="20" > </forms> </authentication> <!-- section denies access files in...
Read more

toolbar - How to add link to user registration inside toobar in admin joomla 3 custom component -

im trying write component admin side , need add user creation. dont want duplicate existing codes add link. know how add task not simple link toolbar. thx help. if wanting button opens new user screen can try in component: $bar = jtoolbar::getinstance('toolbar'); $bar->appendbutton('link', 'users', 'new user','index.php?option=com_users&task=user.add'); you need add view.html.php file wherever you're creating toolbar. is need?
Read more

How to use Authorization & Authentication in Asp.net, C#? -

i using roll management , trying give page , folder access according user or user group, using server created ad group user authentication. i have default1.aspx page default , subdir1 folder give different access separate user group i using below logic in web.config. <location path="subdir1"> <system.web> <authorization> <allow users ="?" /> </authorization> </system.web> </location> i facing problem provide same access 2 or more directory same user should have provide allow user code twice both folder? i can use logic repeating value folder want access providing in 1 logic. i have got answer configure folder/page access, have make different access shown below.. configure access specific file , folder, set forms-based authentication. request page in application redirected logon.aspx automatically. in web.config file, done following code. this code grants user...
Read more