![]() ![]() Or you can use a not so nice but simple work-a-round and execute your shell script in the pig itself and save it's result in a variable, and using that. By using OOZIE project, users of Hadoop can define different actions/jobs and the inter-dependency between the jobs. Understand big data, challenges, distributed environment. From within the workflow definition, the output of an Shell action node is accessible via the String action:output(String node, String key) function (Refer to section '4.2.6 Action EL Functions')." It also covers various Eco-Systems (Hive, Pig, Sqoop, Flume). It also supports system specific jobs such as Java coding. The Shell command output must be in Java Properties file format and it must not exceed 2KB. Oozie is integrated with all the tools in the Hadoop ecosystem - MapReduce, PIG, Hive, Sqoop. "If the capture-output element is present, it indicates Oozie to capture output of the STDOUT of the shell command execution. (actually you have to capture the output as Samson wrote in the comments) The output data is in a Java Properties format and via this EL This function is only applicable to action nodes that produce output (example -> inputtablename.) if the above method is not advisable then i will remove the shell action here and would rather go directly to pig action inside oozie. Hive pigg oozie projects tasks code#Than just register the jars from there: REGISTER '$LIB_PATH/my_jar' Answer: Some things to consider are: a) The Oozie coordinator could be triggered by a source code commit sniffed by your pipeline orchestrator (Jenkins, GoCD, TravisCI, TeamCity, Concourse). The reason why i am placing the pig invoke command inside a shell file is because the pig takes too many number of arguments as input like mentioned in previous comment. Support different types of job such as Hadoop Map-Reduce, Pipe, Streaming, Pig, Hive and custom java. More specifically, this includes: XML-based declarative framework to specify a job or a complex workflow of dependent jobs. ![]() OK make the original question clear, how can you pass a parameter form a shell script output. Oozie is a server-based workflow scheduling and coordination system to manage data processing jobs for Apache Hadoop. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |