/tmp files that appear to be deleted when there are no pipelines running

asked 2018-06-18 14:47:18 -0500

Speiec75
66 ●2 ●2 ●6

updated 2018-06-18 14:58:15 -0500

metadaddy

5464 ●26 ●41 ●81 https://about.me/patpa...

We notice that our /tmp directory was filling up and upon further investigation, Data Collector has files in /tmp/xxxx/pipelineid/.... in (deleted) status. In this environment, there are no pipelines running currently.

Are we suppose to be recycling Data Collector on a daily basis?

Is it normal to see /tmp files that appear to be deleted but are still there when you do a lsof -p against the Data Collector pid?

answered 2018-06-18 15:14:26 -0500

metadaddy

5464 ●26 ●41 ●81 https://about.me/patpa...

You should absolutely not need to recycle Data Collector for this.

What sort of files are they? Any clues from the filenames? Is it possible to use the file command to see what sort of data they contain? Failing that, what sort of operations are you doing in your pipelines? I can see in the source that Data Collector will create files in /tmp when working with whole file data format, but it should be cleaning up after itself. If you can pin down enough information for us to reproduce the problem, please go ahead and create an issue.

edit flag offensive delete link

Comments

I didn't screen print them out and recycle SDC for other reason.. It was /tmp/xxxx/pipelineID/yyy/oracle stuff.. CDC and other extentions.. I will ask my testers to startup a pipeline or two to see if they reappear..

Speiec75 ( 2018-06-18 15:22:59 -0500 )edit

Thanks - that's really helpful. I've alerted the engineer responsible for the Oracle CDC origin.

metadaddy ( 2018-06-18 15:28:15 -0500 )edit

In a different mapr cluster env, sdc filled up /tmp with the following: java 109536 mapr 8393u REG 8,2 134217728 605153378 /tmp/ed0fedbc-4706-11e8-8102-e9d866b6af02/DWSSIDW001incrpub0454f383-64ed-42f7-86c2-e6584e0a1d98/OracleCDCClient_01/7.39.17890/db-7030326246071000779/dbFil

Speiec75 ( 2018-06-29 07:17:03 -0500 )edit

the process is doing a log miner collection into a stream.

Speiec75 ( 2018-06-29 08:23:11 -0500 )edit

Hi @metadaddy. Any way to get some feedback on this issue? A user in South Africa ran into the same issue on version 3.4. He also posted the issue on Jira at https://issues.streamsets.com/browse/SDC-10539

Nick ( 2019-01-17 08:42:47 -0500 )edit

see more comments

/tmp files that appear to be deleted when there are no pipelines running

1 Answer

Comments

Question Tools

Stats

Related questions

/tmp files that appear to be deleted when there are no pipelines running edit

1 Answer

Comments

Question Tools

Stats

Related questions

/tmp files that appear to be deleted when there are no pipelines running