post shell executor not working
I have a shell script which when placed in pre shell executor is working. When the same script is placed in post completion, it is not working.
Details:
- source : sftp whole zip file
- target: hdfs whole zip file
- pipeline finisher executor to stop pipeline when job completes.
- shell executor - script to copy hdfs target file to another hdfs location.
Job is completing successfully without error but copy not working
2018-08-10 21:46:11,343 INFO Starting memory collector for HadoopFS_01
2018-08-10 21:46:11,344 INFO Starting memory collector for PipelineFinisherExecutor_01
2018-08-10 21:46:11,346 INFO Starting memory collector for Shell_01
2018-08-10 21:46:25,818 INFO Destroying pipeline with reason=FINISHED
not even executing the shell script. i put some echo stmts for testing. they are not displaying.
What events are you using - pipeline start/stop or events from the origin/destination? How are you configuring the environment variables for the shell executor? You might be referencing some fields that are present in one context, but not the other.
i tried stop event and event from destination. start event is working but not any stop events. it is exactly same script. log info attached above