Ask Your Question
0

Can I use Kafka stages Azure Event Hub that are Kafka enabled?

asked 2020-07-08 23:22:45 -0500

rupal gravatar image

Azure Event Hub is kafka compatible. https://docs.microsoft.com/en-us/azur... Does StreamSets support using Kafka connectors for kafka-enabled Azure Event Hubs?

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted
0

answered 2020-07-08 23:48:11 -0500

rupal gravatar image

Yes, the Kafka stages in Data Collector work with kafka-enabled Azure Event Hubs. Follow these steps to configure the kafka stages:

Get configuration details from Azure Event Hub:

Using the Azure portal -> Event Hub Namespace -> Event Hub -> Shared Access Policies -> select a policy and copy the 'Connection string–primary key'

Data Collector pipeline configuration:

Add kafka origin/destination.

General tab

Stage library: Use Apache Kafka 1.0 or 2.0

Kafka tab

Broker uri = {EVENTHUBS.NAMESPACE.FQDN}:9093

Topic = {EVENTHUB}

Kafka Configuration: Paste the following in Bulk Edit Mode

[
    {
        "key": "bootstrap.servers",
        "value": "{EVENTHUBS.NAMESPACE.FQDN}:9093"
    },
    {
        "key": "security.protocol",
        "value": "SASL_SSL"
    },
    {
        "key": "sasl.mechanism",
        "value": "PLAIN"
    },
    {
        "key": "sasl.jaas.config",
        "value": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"{Connection string–primary key}\";"
    },
    {
        "key": "serviceName",
        "value": "kafka"
    }
]

Fill in correct values for bootstrap.servers and sasl.jaas.config highlighted in yellow from the information gathered from Azure portal.

Data Format tab

Configure your data format according to your needs.

That's it. Rest of the configurations can be default or tweaked to your liking. Validate and ensure you get a successful connection.

image description

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-07-08 23:22:45 -0500

Seen: 64 times

Last updated: Jul 08