首页 大杂烩正文

Dolphinscheduler/海豚调度器的安装原创

原创
小哥 3年前 (2022-10-22) 阅读数 111 #大杂烩

Dolphinscheduler的安装

1.安装前

1.1 安装节点规划

ips=${ips:-"ds-it-001,ds-it-002,ds-it-003"}                                                                                                                                                                                                             

sshPort=${sshPort:-"22"}                                                                                                                                                                                                                                

masters=${masters:-"ds-it-001,ds-it-002"}                                                                                                                                                                                                              

workers=${workers:-"ds-it-001:default,ds-it-002:default,ds-it-003:default"}                                                                                                                                                                 

alertServer=${alertServer:-"ds-it-002"}                                                                                                                                                                                                                 

apiServers=${apiServers:-"ds-it-002,ds-it-003"}                                                                                                                                                                                                         

installPath=${installPath:-"/data/dolphinscheduler"}                                                                                                                                                                                                    

deployUser=${deployUser:-"dolphinscheduler"}

1.2 软件下载

JDK(1.8.0_144)

Zookeeper(3.8.0)

Postgresql(14.2)

dolphinscheduler(3.0.0-alpha)

二、安装

2.1 创建dolphinscheduler用户

#使用root用户创建用户
useradd dolphinscheduler

#配置sudo免密
sed -i $adolphinscheduler  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL /etc/sudoers
sed -i s/Defaults    requirett/#Defaults    requirett/g /etc/sudoers

#配置免密登录
#在每个节点上生成公钥和私钥。
su - dolphinscheduler
ssh-keygen -t rsa -P  -f ~/.ssh/id_rsa
touch /home/dolphinscheduler/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

#粘贴每个节点生成的公钥authorized_keys
cat > ~/.ssh/authorized_keys

2.2 安装Java

#使用服务器现有的Java配置，但必须确保Java版本是在8以上
export JAVA_HOME=/opt/third/jdk                                                                                              
export JRE_HOME=$JAVA_HOME/jre                                                                                              
export PATH=$PATH:$JAVA_HOME/bin                                                                                            
export CLASSPATH=./:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

#确认Java版本
[root@dp-test-003 ~]# java -version                                                                                         

java version "1.8.0_144"                                                                                                    

Java(TM) SE Runtime Environment (build 1.8.0_144-b01)                                                                       

Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

2.3 安装Zookeeper

Step1.

#在三个节点上分别下载zookeeper安装包
curl -O https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz

Step2.

#解压缩安装包
cd /opt
sudo tar -zxf apache-zookeeper-3.8.0-bin.tar.gz

Step3.

#创建zookeeper软链接
sudo ln -s /opt/apache-zookeeper-3.8.0-bin /opt/zookeeper

Step4.

#修改zoo.cfg配置文件
cd /opt/zookeeper/conf

sudo cp zoo_sample.cfg zoo.cfg

sudo vi zoo.cfg

#修正案如下：
dataDir=/data/zookeeper/zkdata

server.1=10.10.14.134:2888:3888;2181                                                                                        

server.2=10.10.14.212:2888:3888;2181                                                                                        

server.3=10.10.14.211:2888:3888;2181

请注意，下面是以下节点IP请务必匹配，否则启动将失败，并提示以下错误：

2022-05-09 16:12:33,558 [myid:] - ERROR [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):o.a.z.s.q.Leader@325] - Couldnt bind to /10.10.14.211:2888                                                                                            

java.net.BindException: Cannot assign requested address (Bind failed)                                                       

        at java.net.PlainSocketImpl.socketBind(Native Method)                                                               

        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)                                          

        at java.net.ServerSocket.bind(ServerSocket.java:375)                                                                

        at java.net.ServerSocket.bind(ServerSocket.java:329)                                                                

        at org.apache.zookeeper.server.quorum.Leader.createServerSocket(Leader.java:322)                                    

        at org.apache.zookeeper.server.quorum.Leader.lambda$new$0(Leader.java:301)                                          

        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)                                        

        at java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3527)              

        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)                                            

        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)                                     

        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)                                    

        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)                              

        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)                                            

        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)                                           

        at org.apache.zookeeper.server.quorum.Leader.(Leader.java:304)                                                

        at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:1340)                                   

        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1551)

Step5.

 #创建myid文件
 #采取了以下步骤zookeeper集群3在节点上单独执行，myid该文件对应于节点序列号，例如节点。1只需写入1，节点2写入2，节点3写入3
mkdir -p /data/zookeeper/zkdata

echo 1 > /data/zookeeper/zkdata/myid

echo 2 > /data/zookeeper/zkdata/myid

echo 3 > /data/zookeeper/zkdata/myid

Step6.

#启动zookeeper
cd /opt/zookeeper
sudo ./bin/zkServer.sh start

#成功启动后将显示如下
[root@dp-test-001 zookeeper]# ./bin/zkServer.sh start                                                                       

ZooKeeper JMX enabled by default                                                                                            

Using config: /opt/zookeeper/bin/../conf/zoo.cfg                                                                            

Starting zookeeper ... STARTED

2.4 安装PostgreSQL数据库

访问postgresql官网选择下载相应版本

https://www.postgresql.org/download/linux/redhat/

#登录10.10.14.134和10.10.14.211节点，安装数据库软件

# Install the repository RPM:

sudo yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

# Install PostgreSQL:

sudo yum install -y postgresql14-server

#登录主节点10.10.14.134正在正在初始化数据库

sudo /usr/pgsql-14/bin/postgresql-14-setup initdb

sudo systemctl enable postgresql-14

sudo systemctl start postgresql-14

2.5 创建dolphinscheduler数据库

su - postgres

psql

postgres=# create role dolphinscheduler with login password xxxxx;                                                   

CREATE ROLE                                                                                                                 

postgres=# create database dolphinscheduler owner dolphinscheduler;                                                         

CREATE DATABASE

2.6 安装Dolphinscheduler软件

请注意，以下步骤只需要在一个节点上执行。在执行部署脚本时，软件将被复制到配置文件中指定的节点和目录。

#解压软件
cd /opt
curl -O https://dlcdn.apache.org/dolphinscheduler/3.0.0-alpha/apache-dolphinscheduler-3.0.0-alpha-bin.tar.gz
tar -zxf apache-dolphinscheduler-3.0.0-alpha-bin.tar.gz
sudo chown -R dolphinscheduler.dolphinscheduler /opt/apache-dolphinscheduler-3.0.0-alpha-bin

#修改dolphinscheduler_env.sh个人资料，JAVA_HOME、数据库、zookeeper地址
export JAVA_HOME=${JAVA_HOME:-/opt/third/jdk}                                                                               

export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}                                                            

export DATABASE=${DATABASE:-postgresql}                                                                                     

export SPRING_PROFILES_ACTIVE=${DATABASE}                                                                                   

export SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.postgresql.Driver                                                            

export SPRING_DATASOURCE_URL=jdbc:postgresql://dsdb-dp.gz.cvte.cn:5432/dolphinscheduler                                     

export SPRING_DATASOURCE_USERNAME=dolphinscheduler                                                                          

export SPRING_DATASOURCE_PASSWORD=xxxx                                                                           

export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}                                                                                                                                                                                                  

export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}                                                                                                                                    

export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}                                                                            

export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-ds-it-001:2181,ds-it-002:2181,ds-it-003:2181}

#修改install_env.sh
ips=${ips:-"ds-it-001,ds-it-002,ds-it-003"}                                                                                                                                                                                                             

sshPort=${sshPort:-"22"}                                                                                                                                                                                                                                

masters=${masters:-"ds-it-001,ds-it-002"}                                                                                                                                                                                                              

workers=${workers:-"ds-it-001:default,ds-it-002:default,ds-it-003:default"}                                                                                                                                                                 

alertServer=${alertServer:-"ds-it-002"}                                                                                                                                                                                                                 

apiServers=${apiServers:-"ds-it-002,ds-it-003"}                                                                                                                                                                                                         

installPath=${installPath:-"/data/dolphinscheduler"}                                                                                                                                                                                                    

deployUser=${deployUser:-"dolphinscheduler"}  

#正在正在初始化数据库
su - dolphinscheduler

cd /opt/apache-dolphinscheduler-3.0.0-alpha-bin

sh tools/bin/create-schema.sh

#部署dolphinscheduler集群
su - dolphinscheduler

cd /opt/apache-dolphinscheduler-3.0.0-alpha-bin

sh ./bin/install.sh
版权声明

所有资源都来源于爬虫采集,如有侵权请联系我们,我们将立即删除

上一篇：浅谈物料齐套改善原创 下一篇：PostgreSQL开发规范原创
相关文章
最新文章
标签列表
粤ICP备19068341号
所有资源都来源于爬虫采集,如有侵权请联系我们,我们将立即删除
Powered By Z-BlogPHP Theme By 编程老白