Tracking durations of login/logout sessions
Hi Team,
Our VPN and firewall logs continuously stream login and logout events of users connecting via VPN. The events are very verbose and I can identify at least one field in every log line that I can use as unique identifier to track a particular user and also his/her login and logout time.
On a given day more than 100 users would be logging in and out continuously, each one with different durations and connect times.
I have a need to track down the "duration" of each user's session, for metrics and capacity planning purposes. What I am looking for is a way to temporarily store the state information of an individual's login session, so when my streamsets receives a logout event, it can do a lookup and search that user's log-in time and then do some time/date math to compute the duration.
I saw a video on youtube where the author uses redis to store and do in-memory lookups to query state info, but I don't know redis, although I am currently using apache kafka and mysql in my workflow.
Do you have any tutorials, videos, links on sample pipelines that resemble this requirement? I can imagine that a lot of Streamsets users work with raw time stamped log files and have somehow found a way to extract time-based durations in their data.
Any advice on which processors I should be using will be helpful.
Thanks very much in advance.