Flink-on-yarn
摘要:
介紹
官網下載
https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop28-scala_2.11.tgz
flink叢集地址
...
介紹
官網下載
- ofollow,noindex" target="_blank">https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop28-scala_2.11.tgz
flink叢集地址
- ddcve: master
- dddve: slave
- ddeve: slave
解壓
tar -xzvf flink-1.6.1-bin-hadoop28-scala_2.11.tgz mv flink-1.6.1 flink cd flink
修改配置
-
指定master
- vim conf/flink-conf.yaml
jobmanager.rpc.address: ddcve# 指定master
- 指定slaves
echo -e "dddve\nddeve" > conf/slaves
新增yarn配置項
- hadoop/etc/hadoop/yarn-site.xml
<property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>8</value> </property>
啟動flink
./bin/start-cluster.sh
- 輸出結果如下
Starting HA cluster with 2 masters. Starting standalonesession daemon on host ddcvc. Starting standalonesession daemon on host ddcve. Starting taskexecutor daemon on host dddve. Starting taskexecutor daemon on host ddeve.
flink on yarn
以下操作都在slave
- 啟動yarn-session
./bin/yarn-session.sh -d -n 4 -tm 2048 -s 4
- 提交job
# 先在hdfs建立輸入檔案 hadoop fs -copyFromLocal /tmp/passwd /tmp/passwd # 提交 ./bin/flink run ./examples/batch/WordCount.jar --input hdfs:///tmp/passwd
flink with zookeeper
- 修改conf/flink-conf.yaml
high-availability: zookeeper high-availability.storageDir: hdfs:///flink/ha/ high-availability.zookeeper.quorum: ddcva:2181,dddva:2181,ddeva:2181 recovery.zookeeper.path.root: /flink high-availability.zookeeper.client.acl: open state.backend: filesystem
- 修改conf/masters
ddcvc:8081 ddcve:8081