Ask Your Question

Revision history [back]

How do I split fields in log data?

I have the following event data coming out of a Versa SD Wan Manger. The goal is to ingest the data via a syslog feed and then transform it so that is can be sent into an ElasticSearch cluster.

Currently I'm reading a sample file in from a directory with a data format of csv.

The next step I think I need to do is to split the data into the field name and value using the equal sign as a delimiter. From there it would be passed into an index in the ES cluster.

The other issue is that of how to keep the time date and the event log name that is at the front of each record.

2017-11-26T22:36:31+0000 cgnatLog, applianceName=Site1Branch1, tenantName=Customer1, observationTimeMilliseconds=2337165310, flowId=33655871, flowCookie=1511734794, sourceIPv4Address=172.18.101.10, destinationIPv4Address=8.8.8.8, postNATSourceIPv4Address=70.70.5.2, postNATDestinationIPv4Address=8.8.8.8, sourcePort=37190, destinationPort=53, postNAPTsourceTransportPort=45643, postNAPTdestinationTransportPort=53, tenantId=1, vsnId=0, applianceId=0, protocolIdentifier=17, sourceNatPoolName=DIA-Pool-ISPA-Network, destNatPoolName=-, natRuleName=DIA-Rule-Customer1-LAN1-VR-ISPA-Network, natEvent=nat44-sess-create

Is this a reasonable approach and is there a better process that can be used to achieve desired goal?