Ask Your Question
0

Log processed record counts

asked 2019-05-08 15:50:14 -0500

metadaddy gravatar image

updated 2019-05-08 15:50:21 -0500

Is there any way to log the number of records processed at each stage during 1 pipeline run?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-05-08 15:51:33 -0500

metadaddy gravatar image

The data is available via the history REST API. For example, here are the counters for the last run of a given pipeline. I'm using the excellent jq tool to parse the JSON at the command line.

$ curl -s -u admin:admin -H 'X-Requested-By:sdc' http://localhost:18630/rest/v1/pipeline/RedshiftStreamingwithKinesisFirehose537add73-bb16-4358-a26a-a51576dea32b/history | jq -r .[0].metrics | jq .counters
{
  "pipeline.batchCount.counter": {
    "count": 1029
  },
  "pipeline.batchErrorMessages.counter": {
    "count": 0
  },
  "pipeline.batchErrorRecords.counter": {
    "count": 0
  },
  "pipeline.batchInputRecords.counter": {
    "count": 648226
  },
  "pipeline.batchOutputRecords.counter": {
    "count": 648226
  },
  "stage.ExpressionEvaluator_01.errorRecords.counter": {
    "count": 0
  },
  "stage.ExpressionEvaluator_01.inputRecords.counter": {
    "count": 648226
  },
  "stage.ExpressionEvaluator_01.outputRecords.counter": {
    "count": 648226
  },
  "stage.ExpressionEvaluator_01.stageErrors.counter": {
    "count": 0
  },
  "stage.ExpressionEvaluator_01:ExpressionEvaluator_01OutputLane15561338960790.outputRecords.counter": {
    "count": 648226
  },
  "stage.FieldOrder_01.errorRecords.counter": {
    "count": 0
  },
  "stage.FieldOrder_01.inputRecords.counter": {
    "count": 648226
  },
  "stage.FieldOrder_01.outputRecords.counter": {
    "count": 648226
  },
  "stage.FieldOrder_01.stageErrors.counter": {
    "count": 0
  },
  "stage.FieldOrder_01:FieldOrder_01OutputLane15561351879260.outputRecords.counter": {
    "count": 648226
  },
  "stage.FieldTypeConverter_01.errorRecords.counter": {
    "count": 0
  },
  "stage.FieldTypeConverter_01.inputRecords.counter": {
    "count": 648226
  },
  "stage.FieldTypeConverter_01.outputRecords.counter": {
    "count": 648226
  },
  "stage.FieldTypeConverter_01.stageErrors.counter": {
    "count": 0
  },
  "stage.FieldTypeConverter_01:FieldTypeConverter_01OutputLane15560499048280.outputRecords.counter": {
    "count": 648226
  },
  "stage.KinesisFirehose_01.errorRecords.counter": {
    "count": 0
  },
  "stage.KinesisFirehose_01.inputRecords.counter": {
    "count": 648226
  },
  "stage.KinesisFirehose_01.outputRecords.counter": {
    "count": 648226
  },
  "stage.KinesisFirehose_01.stageErrors.counter": {
    "count": 0
  },
  "stage.MySQLBinaryLog_01.errorRecords.counter": {
    "count": 0
  },
  "stage.MySQLBinaryLog_01.inputRecords.counter": {
    "count": 0
  },
  "stage.MySQLBinaryLog_01.outputRecords.counter": {
    "count": 648226
  },
  "stage.MySQLBinaryLog_01.stageErrors.counter": {
    "count": 0
  },
  "stage.MySQLBinaryLog_01:MySQLBinaryLog_01OutputLane15561313696850.outputRecords.counter": {
    "count": 648226
  },
  "stage.StreamSelector_01.errorRecords.counter": {
    "count": 0
  },
  "stage.StreamSelector_01.inputRecords.counter": {
    "count": 648226
  },
  "stage.StreamSelector_01.outputRecords.counter": {
    "count": 648226
  },
  "stage.StreamSelector_01.stageErrors.counter": {
    "count": 0
  },
  "stage.StreamSelector_01:StreamSelector_01OutputLane1556133811620.outputRecords.counter": {
    "count": 0
  },
  "stage.StreamSelector_01:StreamSelector_01OutputLane1556133816638.outputRecords.counter": {
    "count": 648226
  },
  "stage.Trash_01.errorRecords.counter": {
    "count": 0
  },
  "stage.Trash_01.inputRecords.counter": {
    "count": 0
  },
  "stage.Trash_01.outputRecords.counter": {
    "count": 0
  },
  "stage.Trash_01.stageErrors.counter": {
    "count": 0
  }
}
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-05-08 15:50:14 -0500

Seen: 192 times

Last updated: May 08