GitXplorerGitXplorer
o

KafkaFlinkElasticsearch

public
0 stars
0 forks
0 issues

Commits

List of commits on branch master.
Unverified
458e1af9a0e9610772949d5da9b33ee3eb71104d

KafkaFlinkElasticsearch update

committed 6 years ago
Unverified
69fcc2c791105fd2a79c794a30fd3d6dfede49f9

KafkaFlinkElasticsearch

committed 6 years ago
Unverified
ddf12837f06d88883baa6a4056a15a60ced0c79d

KafkaFlinkElasticsearch

committed 6 years ago
Unverified
a3c21b8423e0be889f3e10c0aeda6fce1fbc3b8c

update

committed 6 years ago
Unverified
4351af98006e0bb4ec3b93813db3c639f8804d79

update

committed 6 years ago
Unverified
345e9e4ea38859e80ad5a3efe7c700cfcaec4dc3

first Commit

committed 6 years ago

README

The README file for this repository.

Kafka Flink Elasticsearch-6.4.2

Thanks to keiraqz. Updated version of the keiraqz's project for Elasticsearch-6.4.2 and Kafka 0.11. https://github.com/keiraqz/KafkaFlinkElastic

  • Build the project
bıild : mvn clean install 
run   : mvn exec:java -Dexec.mainClass=viper.KafkaFlinkElastic
```

Flink 2.11 | Kafka 0.11 | Elasticsearch 6.4.2

  • Prepare Kafka & Elasticsearch

    • Assume you have Kafka and Elasticsearch installed on your local machine

      • Create a kafka topic:
       /usr/local/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic viper_test
      
      • Create Elasticsearch index & doctype
       # create viper-test index
       curl -XPUT 'http://localhost:9200/viper-test/' -d '{
           "settings" : {
               "index" : {
                   "number_of_shards" : 1, 
                   "number_of_replicas" : 0
               }
           }
       }'
       
       # put mapping for viper-log doctype
       curl -XPUT 'localhost:9200/viper-test/_mapping/viper-log' -d '{
       	  "properties": {
       		    "ip": {
       		      "type": "string",
       		      "index": "not_analyzed"
       		    },
       		    "info": {
       		        "type": "string"
       		    }
       	  }
       }'
      
  • Flink & Kafka Connector

    • Maven dependency
     <dependency>
     		<groupId>org.apache.flink</groupId>
     		<artifactId>flink-connector-kafka-0.11_2.11</artifactId>
     		<version>${flink.version}</version>
     </dependency>	```
     
     
    
  • Flink & Elasticsearch Connector

    • Maven dependency
     <dependency>
     		<groupId>org.apache.flink</groupId>
     		<artifactId>flink-connector-elasticsearch6_2.11</artifactId>
     		<version>1.6.0</version>
     </dependency>
    
  • Telegram get your own token and chat id and change the urlString in sendToTelegram method.