Ask Your Question

How to remove special characters from a field name?

asked 2017-06-27 12:56:50 -0500

rupal gravatar image

updated 2018-07-31 15:31:58 -0500

jeff gravatar image

How can a pipeline remove special characters or convert all special characters to something else in all field names?

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2019-09-13 14:45:55 -0500

jeff gravatar image

As of Data Collector 3.8.0, you can also use the Field Mapper Processor to accomplish this. See the example in the documentation.

edit flag offensive delete link more

answered 2017-06-27 14:08:42 -0500

rupal gravatar image

updated 2017-06-27 17:49:26 -0500

LC gravatar image

Try using a Field Renamer processor and provide a regex pattern. It may become cumbersome with a Field Renamer and may be best using a script to do this. For example, you can use the groovy script below with the Groovy Evaluator to convert all special characters in an entire field to underscores:

import java.util.regex.*
Pattern p = Pattern.compile("[^a-zA-Z0-9]");
for (record in records) {
  try {
    def toRemove = []
    def toAdd = [:]
    record.value.each { key, value ->
      if (p.matcher(key).find()) {
        toAdd.put(key.replaceAll("[^A-Za-z0-9]", "_"), value)

    toRemove.each { field -> record.value.remove(field) }
    toAdd.each { key, value -> record.value.put(key, value) }

  } catch (e) {
    // Write a record to the error pipeline
    log.error(e.toString(), e)
    error.write(record, e.toString())
edit flag offensive delete link more
Login/Signup to Answer

Question Tools


Asked: 2017-06-27 12:56:50 -0500

Seen: 1,915 times

Last updated: Sep 13 '19