The Write Text to a Kafka Topic Output Connector can be used to write event data, adapted and formatted as delimited text, to an Apache Kafka Topic. For more information about getting started with Apache Kafka, see Apache Kafka Introduction.
Usage notes
- Use this output connector to write data, adapted and formatted as delimited text, to a Kafka Topic. This output connector is a producer to Kafka.
- This output connector pairs the Text Outbound Adapter with the Kafka Outbound Transport.
- A Message Separator and an Attribute Separator are required to format event records as delimited text before writing to Kafka. The Message Separator indicates the character which identifies the end of a data record; the default is \n (newline). The Attribute Separator specifies the character used to separate one attribute from another in a single line of text; the default is , (comma). Any normal ASCII character, specified as the character or Unicode value, can be used as a message or attribute separator.
- The Kafka Inbound Transport supports TLS 1.2 and SASL security protocols for authenticating with a Kafka cluster or broker.
Parameters
| Parameter | Description |
|---|---|
Name | A descriptive name for the input connector used for reference in GeoEvent Manager. |
Override with Custom Kafka Properties | Specify whether to override the default GeoEvent Server Kafkaclient properties. The default is No.
|
Kafka Bootstrap Servers (Conditional) | A list of hostname:port pairs used to establish the initial connection to the Kafka cluster. Hostname:port pairs must be comma-separated, for example:
|
Registered Folder for the Kafka Properties File (Conditional) | The folder registered with GeoEvent Server that contains the Kafka .properties file. The Kafka .properties file defines the custom Kafka properties when Override with Custom Kafka Properties is set to Yes. Ensure that the folder registered with GeoEvent Server is the full path to where the Kafka .properties file is located. |
Kafka Properties File Name (Conditional) | The name of the Kafka .properties file that contains the custom Kafka properties for client configuration. The name of the file should be specified without the .properties extension.
|
Topic Name | The name of the Kafka topic to publish data to.
Note:The Kafka Outbound Transport does not support publishing data to multiple topics. |
Enable Exactly Once Delivery | Specifies whether exactly-once semantics should be honored when writing to the topic. See Exactly Once Semantics for more information. Default is Yes.
Note:Ensure the Kafka cluster supports exactly once semantics before continuing with the Enable Exactly Once Delivery property. Enabling exactly once semantics comes with a performance cost. For more information, see Producer Configs as related to acknowledgments. |
Event Separator | A single literal character which indicates the end of an event data record. Unicode values may be used to specify a character delimiter. The character should not be enclosed in quotes. A newline (\n) is a common end-of-record delimiter. |
Field Separator | A single literal character used to separate one attribute value from another in a message. Unicode values may be used to specify a character delimiter. The character should not be enclosed in quotes. A comma (,) is a common attribute delimiter. |
Output Date Format | Specifies the format of date/time values written to the file. The available output date formats are ISO 8601 Format or a Custom Format. The default is ISO 8601 Format.
|
Custom Date Format | The custom format for date/time values written to the file. The custom date format should be constructed using the Java SimpleDateFormat class convention. For more information, see the SimpleDateFormat. |
Language for Number Formatting | The locale identifier (ID) used for locale-sensitive behavior when formatting numbers from data values. The default is the locale of the machine GeoEvent Server is installed on. For more information, see Java Supported Locales. |
Authentication Required | Specifies whether the connection to the Kafka cluster, or Kafka broker, requires authentication. The default is No.
|
Authenticate Using (Conditional) | Specifies the security protocol that is used to secure the Kafka cluster. Available security protocols include TLS 1.2 and SASL.
Note:When using Kerberos, ensure the operating system user account running ArcGIS GeoEvent Server has read-access to the keytab file in the Kerberos setup/configuration. Property is shown when Authentication Required is set to Yes. |
Registered Folder for Credential File (Conditional) | The folder registered with GeoEvent Server that contains the Kafka cluster's PKI file (x509 certificate). Ensure that the folder registered with GeoEvent Server is the full path to where the Kafka cluster's certificate is located. |
Credential Configuration File (Conditional) | The name of the Kafka cluster's PKI file (x509 certificate). The certificate and its associated private key must be stored in the PKCS#12 format, which is represented by a file with either the .p12 or .pfx extension. Provide the name of the file in addition to the extension.
Note:Only the certificate file name and extension are supported for this parameter. Relative paths to the certificate should not be specified in this parameter. Register the full path to the certificate file using the Registered Folder for Credential File parameter. Property is shown when Authentication Required is set to Yes. Properly is applicable to TLS 1.2 only. |
Keystore Password (Conditional) | The password for the Kafka cluster's PKI file (x509 certificate). Also known as the certificate's private key. |
SASL Authentication Type (Conditional) | Specifies the type of SASL authentication mechanism supported by the Kafka cluster. Available SASL authentication types include SASL GSSAPI (Kerberos) and SASL PLAIN.
|
Kerberos Principal (Conditional) | The Kerberos principal for the specific user. For example:
|
Use Key Tab (Conditional) | Indicates whether to use the keytab in the Kerberos settings. The default is Yes.
|
Store Key (Conditional) | Indicates whether to store the key in the Kerberos settings. The default is Yes.
|
Username (Conditional) | Specifies the username used to authenticate with the Kafka cluster. Also known as a connection string with certain cloud providers. Refer to the documentation of the chosen cloud provider for correct syntax. |
Password (Conditional) | Specifies the password used to authenticate with the Kafka cluster. Refer to the documentation of the chosen cloud provider for the correct syntax. |
Considerations and limitations
- The Write Text to a Kafka Topic output connector is a producer to Kafka. Apply the same considerations to this output connector as would be required for any other external producer to Kafka.