Ask Your Question
1

Why cant field renamer use str:replace as Target Field Expression?

asked 2019-08-27 00:27:38 -0600

Challenge Addict gravatar image

updated 2019-08-29 23:41:36 -0600

The documentation for the Field Renamer under 'Change Capitalization' seems to suggest that its possible to use functions as part of a Target Field Expression.

My source field expression is /'(.*-.*)' and my target field expression is /${str:replace("$1","-","_")}:

Renamer Configuration

However, I receive the error:

Record1-Error FIELD_RENAMER_01 - Target Fields '/a-aa, /bb-b, /c-c-c' cannot be overwritten for record 'IngestMe.csv::5578'

Screenshot: Renamer Error

When I set the 'Target Field Already Exists' setting to 'REPLACE', the pipeline simply removes all fields containing a hyphen ('-'). Why? What is my target field expression evaluating to?

You can download this sample pipeline here.

edit retag flag offensive close merge delete

Comments

1

Modded you up, @Challenge Addict, so you can post screenshots. Note - you can't attach JSON - just images, for some reason.

metadaddy gravatar imagemetadaddy ( 2019-08-29 01:17:09 -0600 )edit

What version of Data Collector are you using, @Challenge Addict? I just tried your exact pipeline and it works as expected - fields are renamed to b_bb and cc_c.

metadaddy gravatar imagemetadaddy ( 2019-09-02 22:50:34 -0600 )edit

Hi @metadaddy, sorry, I should have mentioned that in my post. I am using StreamSets Data Collector 3.3.1. Is this version too old?

Challenge Addict gravatar imageChallenge Addict ( 2019-09-03 20:33:22 -0600 )edit
1

Please do try a newer version. I imported the pipeline you shared and it runs perfectly fine in the current nightly (3.11.0, in development). There were a couple of bugfixes for this stage, but I can't tell which one might be involved here.

jeff gravatar imagejeff ( 2019-09-09 16:54:11 -0600 )edit
1

Actually I am thinking https://issues.streamsets.com/browse/SDC-10463 is the underlying bug here. You essentially replaced `aaa` with itself (since it didn't contain any dash), which triggers that error.

jeff gravatar imagejeff ( 2019-09-09 16:55:52 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
-1

answered 2019-08-27 12:02:21 -0600

jeff gravatar image

Can you reproduce the problem in a simple pipeline with Dev Raw origin, then renamer, then Trash destination? I just tried creating such a pipeline and wasn't able to reproduce this error, using the configuration you specified. What is the structure of your record going into the renamer (full record)?

edit flag offensive delete link more

Comments

Hi Jeff, I've created a pipeline as you requested and the problem still manifests itself in the same way. I've linked it in my question above.

Challenge Addict gravatar imageChallenge Addict ( 2019-08-29 01:13:30 -0600 )edit

I've now added screenshots on top of all that. Hope it should be quite clear by now?

Challenge Addict gravatar imageChallenge Addict ( 2019-09-02 18:54:03 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-08-27 00:27:38 -0600

Seen: 148 times

Last updated: Aug 29