azure data lake store destination fails to close file with 401

asked 2018-10-12 05:53:04 -0500

Non gravatar image

Hi, When defining an azure data lake store as destination, it seems like that streamsets is not able to refresh the azure AD token after it expires. (default lifetime is one hour) As a result the pipeline is not able to close the file when the idle time is reached and encounters a 401 http authentication error. (see log below) How can i configure streamsets to refresh token after expiration?

I'm using streamsets version 3.5.0

2018-10-12 10:05:37,287 [user:*admin] [pipeline:] [runner:] [thread:Data Lake Idle Close Thread] ERROR DataLakeDataGenerator - Error while attempting to close /msg/avro_data/2018-10-12/_tmp_sdc-36fec689-cb0d-11e8-806e-23720372960e-36fec689-cb0d-11e8-806e-23720372960e
com.microsoft.azure.datalake.store.ADLException: Error appending to file /msg/avro_data/2018-10-12/_tmp_sdc-36fec689-cb0d-11e8-806e-23720372960e-36fec689-cb0d-11e8-806e-23720372960e-FROMKAFKAkopfrechnerrawtopicTOAzureDataLake1fca9705-3414-4eb1-a415-62935d852067-0
Operation APPEND failed with HTTP401 : null
Last encountered exception thrown after 5 tries. [HTTP401(null),HTTP401(null),HTTP401(null),HTTP401(null),HTTP401(null)]
 [ServerRequestId:dae53b26-7adf-4bd5-bb94-1c4df337622a]
    at com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1122)
    at com.microsoft.azure.datalake.store.ADLFileOutputStream.flush(ADLFileOutputStream.java:180)
    at com.microsoft.azure.datalake.store.ADLFileOutputStream.close(ADLFileOutputStream.java:216)
    at org.apache.commons.io.output.ProxyOutputStream.close(ProxyOutputStream.java:117)
    at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
    at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
    at org.apache.avro.file.DataFileWriter.close(DataFileWriter.java:434)
    at com.streamsets.pipeline.lib.generator.avro.BaseAvroDataGenerator.close(BaseAvroDataGenerator.java:213)
edit retag flag offensive close merge delete