Ask Your Question

How do I split fields in log data?

asked 2019-06-28 19:02:16 -0600

metadaddy gravatar image

I have the following event data coming out of a Versa SD Wan Manger. The goal is to ingest the data via a syslog feed and then transform it so that is can be sent into an ElasticSearch cluster.

Currently I'm reading a sample file in from a directory with a data format of csv.

The next step I think I need to do is to split the data into the field name and value using the equal sign as a delimiter. From there it would be passed into an index in the ES cluster.

The other issue is that of how to keep the time date and the event log name that is at the front of each record.

2017-11-26T22:36:31+0000 cgnatLog, applianceName=Site1Branch1, tenantName=Customer1, observationTimeMilliseconds=2337165310, flowId=33655871, flowCookie=1511734794, sourceIPv4Address=, destinationIPv4Address=, postNATSourceIPv4Address=, postNATDestinationIPv4Address=, sourcePort=37190, destinationPort=53, postNAPTsourceTransportPort=45643, postNAPTdestinationTransportPort=53, tenantId=1, vsnId=0, applianceId=0, protocolIdentifier=17, sourceNatPoolName=DIA-Pool-ISPA-Network, destNatPoolName=-, natRuleName=DIA-Rule-Customer1-LAN1-VR-ISPA-Network, natEvent=nat44-sess-create

Is this a reasonable approach and is there a better process that can be used to achieve desired goal?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-06-28 19:03:32 -0600

metadaddy gravatar image

A better approach would be to use Log data format and with a Grok pattern to parse the data. This works for me with your data:

Grok Pattern Definition: NATNAME (?:[0-9A-Za-z-]+)

Grok Pattern: %{TIMESTAMP_ISO8601:timestamp} %{WORD:logname}, applianceName=%{WORD:applianceName}, tenantName=%{WORD:tenantName}, observationTimeMilliseconds=%{POSINT:observationTimeMilliseconds}, flowId=%{POSINT:flowId}, flowCookie=%{POSINT:flowCookie}, sourceIPv4Address=%{IPV4:sourceIPv4Address}, destinationIPv4Address=%{IPV4:destinationIPv4Address}, postNATSourceIPv4Address=%{IPV4:postNATSourceIPv4Address}, postNATDestinationIPv4Address=%{IPV4:postNATDestinationIPv4Address}, sourcePort=%{POSINT:sourcePort}, destinationPort=%{POSINT:destinationPort}, postNAPTsourceTransportPort=%{POSINT:postNAPTsourceTransportPort}, postNAPTdestinationTransportPort=%{POSINT:postNAPTdestinationTransportPort}, tenantId=%{NONNEGINT:tenantId}, vsnId=%{NONNEGINT:vsnId}, applianceId=%{NONNEGINT:applianceId}, protocolIdentifier=%{NONNEGINT:protocolIdentifier}, sourceNatPoolName=%{NATNAME:sourceNatPoolName}, destNatPoolName=%{NATNAME:destNatPoolName}, natRuleName=%{NATNAME:natRuleName}, natEvent=%{NATNAME:natEvent}

image description

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-06-28 19:02:16 -0600

Seen: 320 times

Last updated: Jun 28 '19