|
centos7.2 安装hdp2.6.1步骤 环境： Operating System: CentOS Linux release 7.2.1511 (Core) Kernel: Linux 3.10.0-327.el7.x86_64 HDP:2.6.1.0 Ambari:ambari-server-2.5.0.3 下载地址： ambari-2.5.0.3 tar包下载地址：http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos7.tar.gz HDP-2.6.1.0 tar包下载地址：http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0/HDP-2.6.1.0-centos7-rpm.tar.gzMD5:HDP-2.6.1.0-centos7-rpm.tar.gz: 48 3F C7 17 19 4A 83 94 82 48 70 7C D4 85 62 14 HDP-UTILS 2.6.10 tar包下载地址：http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gzMD5:HDP-UTILS-1.1.0.21-centos7.tar.gz: B6 1E 3F 3D D9 10 51 8D 86 3D 44 FC 59 15 57 52 0.所有主机安装jdk并替换jce文件(若启用kerberos) # tar -zxvf /data/jdk-8u111-linux-x64.tar.gz -C /usr/ # unzip /data/UnlimitedJCEPolicyJDK8.zip # cp -f /data/UnlimitedJCEPolicyJDK8/local_policy.jar /usr/jdk1.8.0_111/jre/lib/security/ # cp -f /data/UnlimitedJCEPolicyJDK8/US_export_policy.jar /usr/jdk1.8.0_111/jre/lib/security/ # vi /etc/profile 末尾添加 export JAVA_HOME=/usr/jdk1.8.0_111 export PATH=$JAVA_HOME/bin:$PATH #. /etc/profile 1.ssh信任配置(略) 2.ntp配置 2.1 安装并开启ntp yum install -y ntp 2.2 开启ntp systemctl is-enabled ntpd 2.3 vi /etc/ntp.conf 注释其他server，添加本地ntp服务器 server x.x.x.x iburst 2.4 systemctl enable ntpd systemctl start ntpd systemctl status ntpd.service 检查服务是否正常 3.主机名配置 hostnamectl set-hostname server_name 查看主机信息 [root@eva0 yum.repos.d]# hostnamectl Static hostname: eva0 Icon name: computer-vm Chassis: vm Machine ID: b5b4a6e010034d50axxxxxxxxxxxxxxx Boot ID: 818af0fc48c64931bxxxxxxxxxxxxxxx Virtualization: vmware Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.10.0-514.10.2.el7.x86_64 Architecture: x86-64 4.配置/etc/hosts，不要更改下述行，否则会引起各种问题 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 eva0 x.x.x.x eva1 x.x.x.x eva2 x.x.x.x 之后hostname -f检查是否配置成功 5.所有主机关闭防火墙 # systemctl disable firewalld # service firewalld stop 6.关闭selinux，packagekit(如果已启用) vi /etc/yum/pluginconf.d/refresh-packagekit.conf enabled=0 7.更改umask # umask 0022 # echo umask 0022 >> /etc/profile 8. 建立HDP local repository 8.0 启动http服务 # systemctl start httpd.service # systemctl status httpd.service 8.1 下载并解压HDP软件包 tar -zxvf HDP-2.6.1.0-centos7-rpm.tar.gz -C /var/www/html mkdir -p /var/www/html/HDP-UTIL tar -zxvf HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/HDP-UTIL/ tar -zxvf ambari-2.5.0.3-centos7.tar.gz -C /var/www/html 检查通过http是否可访问到资源 Ambari Base URL http://1eva0/ambari/centos7/ambari/ HDP Base URL http://1eva0/HDP/centos7/ HDP-UTILS Base URL http://1eva0/HDP-UTIL/ 8.2 安装createrepo包 yum install createrepo 8.3 创建repocreaterepo /var/www/html/ambari/centos7/ createrepo /var/www/html/HDP/centos7 createrepo /var/www/html/HDP-UTIL (注意createrepo的目录) 8.4 修改*.repo文件 # more /etc/yum.repos.d/ambari.repo #VERSION_NUMBER=2.5.1.0 [ambari-2.5.1.0] name=ambari-2.5.1.0 baseurl=http://eva0ambari/centos7/ambari gpgcheck=0 #gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/RPM-GPG-KEY/RPM-GPG -KEY-Jenkins enabled=1 priority=1 # more /etc/yum.repos.d/HDP.repo #VERSION_NUMBER=2.6.1.0-129 [HDP-2.6.1.0] name=HDP Version - HDP-2.6.1.0 baseurl=http://eva0HDP/centos7 gpgcheck=0 gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0/RPM-GPG-KEY/RPM-GPG-KEY -Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.21] name=HDP-UTILS Version - HDP-UTILS-1.1.0.21 baseurl=http://eva0HDP-UTIL gpgcheck=0 gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0/RPM-GPG-KEY/RPM-GPG-KEY -Jenkins enabled=1 priority=1 9 建立centos7 local repository 9.0 挂载软件iso镜像 mkdir -p /var/www/html/os7 mount -o loop /data/centos7.iso /var/www/html/os7 9.1 制作repository 方法一： mkdir -p /var/www/html/centos7 直接将Packages目录及其下的软件包copy到/var/www/html/centos7 方法二： mkdir -p /data/centos7 将Packages目录及其下的软件包copy到/data/centos7 ln -s /data/centos7 /var/www/html/centos7 9.2 创建repo(以方法一为例) createrepo /var/www/html/centos7 9.3修改repo文件 #vi /etc/yum.repos.d/CentOS-Base.repo [base] name=CentOS-$releasever - Base #mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra baseurl=http://133.37.126.170/centos7/Packages/ gpgcheck=0 #gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 enable=1 priority=1 注意enable=1，gpgcheck设置为0 10. 检查当前repository库 先修改repo目录的权限 chmod -R 777 /var/www/html/centos7 chmod -R 777 /var/www/html/ambari chmod -R 777 /var/www/html/HDP chmod -R 777 /var/www/html/HDP-UTIL # yum repolist 已加载插件：fastestmirror, langpacks Loading mirror speeds from cached hostfile 源标识 源名称 状态 HDP-2.6.1.0 HDP Version - HDP-2.6.1.0 232 HDP-UTILS-1.1.0.21 HDP-UTILS Version - HDP-UTILS-1.1.0.21 64 ambari-2.5.0.3 ambari Version - ambari-2.5.0.3 12 base CentOS-7 - Base 3,723 repolist: 4,031 重要：检查无误后，将三个repo文件复制到所有节点上 并检查其他节点的/etc/yum.conf文件中是否设置了proxy代理，如果有应暂时取消，以避免可能无法访问到做repository的主机。 11.安装ambari server 选择eva0主机安装： 注意ambari-server依赖postgresql，当前2.5.0.3版本需要postgresql-libs(x86-64) = 9.2.13-1.el7_1，如果os中已经安装不同版本的pg，先卸载再安装指定版本 [root@eva0 ~]# yum erase postgresql-libs-9.2.18-1.el7.x86_64 [root@eva0 ~]# yum install ambari-server 12.安装mysql服务器 安装 yum localinstall MySQL-server-5.6.14-1.el6.x86_64.rpm yum localinstall MySQL-devel-5.6.14-1.el6.x86_64.rpm yum localinstall MySQL-client-5.6.14-1.el6.x86_64.rpm 配置 vi /etc/my.cnf [mysqld] #basedir=/usr/share/mysql datadir=/data/mysql/data socket=/var/lib/mysql/mysql.sock user=mysql # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 skip-name-resolve [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/lib/mysql/eva0.pid [client] port=3306 socket=/var/lib/mysql/mysql.sock 启动 mysql_install_db --user=mysql --defaults-file=/etc/my.cnf ./mysql.server start mysqladmin -u root password 'yyyyy' 13.设置ambari-server # ambari-server setup 使用默认用户 Customize user account for ambari-server daemon [y/n] (n)? 使用自行安装的jdk Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 输入JAVA_HOME Path to JAVA_HOME: /usr/jdk1.8.0_111 至于ambari的元数据存储位置，可以自定义，如选择mysql，也可直接使用默认的pg，测试环境建议直接默认， Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 3 Hostname (localhost): Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): 所有设置采用默认 到此步时不要继续，先完成下述两步骤： 13.1 将mysql的jdbc 的dirver jar放到/usr/share/java目录下，并在ambari.properties文件中添加server.jdbc.driver.path=[path/to/custom_jdbc_driver]属性 (可从https://dev.mysql.com/downloads/connector/j/ 网站下载) 新开窗口执行： copy driver到指定目录 tar -zxvf /data/mysql-connector-java-5.1.42.tar.gz cp /data/mysql-connector-java-5.1.4/mysql-connector-java-5.1.42-bin.jar /usr/share/java/ 修改ambari配置文件 vi /etc/ambari-server/conf/ambari.properties 添加 server.jdbc.driver.path=/usr/share/java/mysql-connector-java-5.1.42-bin.jar 13.2 运行ambari配置sql脚本 新开窗口执行： [root@eva0 ~]# mysql -uroot -pyyyyy </var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql 注：如果报错，可根据脚本内容先创建db，用户，修改use库的行并取消注释 -------------------------------------- # CREATE DATABASE `ambari` /*!40100 DEFAULT CHARACTER SET utf8 */; # CREATE USER 'ambari' IDENTIFIED BY 'bigdata'; USE ambari; ---------------------------------------- 13.3 回到原窗口，继续配置过程，即可成功完成配置 Configuring ambari database... Copying JDBC drivers to server resources... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? Extracting system views... ............ Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully. 14.启动ambari-server，通过其安装配置hdp软件包 ambari-server start 15启动报错解决： 15.01查看ambari日志定位错误 ava.sql.SQLException: Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED. 实际与mysql配置有关，需要更改bin-log format 查看当前binlogformat mysql> select @@binlog_format; +-----------------+ | @@binlog_format | +-----------------+ | STATEMENT | +-----------------+ 1 row in set (0.01 sec) 当前服务修改，mysql重启后失效 SET SESSION binlog_format = 'MIXED' ; SET GLOBAL binlog_format = 'ROW'; 永久修改 vi /etc/my.cnf 添加 binlog-format = ROW 15.02 ambari用户无读取db的权限 ERROR [main] DBAccessorImpl:117 - Error while creating database accessor com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Access denied for user 'ambari'@'%' to database 'ambari' am # mysql -uroot -p Enter password: mysql> grant all privileges on ambari.* to 'ambari'@'%' ; mysql> flush privileges; 16 登陆ambari web 进行配置 初始密码 admin/admin 若忘记密码，可在后台更改： mysql> use ambari; mysql> update users set user_password=PASSWORD('12345') where user_id=1; mysql> flush privileges; 17配置中可能存在的问题 17.1 配置hive metastore的存储数据库时指定已有mysql数据库， 需先在ambari-server上执行： # ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-5.1.42-bin.jar 再在mysql中执行以下命令创建hive对应db，用户和赋权 mysql> create database hive; mysql> create user hive identified by 'yyyyy'; Query OK, 0 rows affected (0.01 sec) mysql> grant all privileges on hive.* to 'hive'@'%'; Query OK, 0 rows affected (0.01 sec) mysql> flush privileges; Query OK, 0 rows affected (0.02 sec) mysql> update mysql.user set password=PASSWORD('yyyyy') where user='hive'; Query OK, 2 rows affected (0.01 sec) Rows matched: 3 Changed: 2 Warnings: 0 mysql> flush privileges; Query OK, 0 rows affected (0.01 sec) 17.2 ambari自动安装hadoop软件时失败，日志显示libtirpc-devel缺失：如果机器上已经有libtirpc软件包，找到对应版本的 libtirpc-devel安装即可 如果找不到对应版本建议卸载现有libtirpc包，重装版本匹配的libtirpc和libtirpc-devel # rpm -qa|grep libtirpc libtirpc-0.2.4-0.6.el7.x86_64 非对应版本无法安装 # rpm -ivh libtirpc-devel-0.2.4-0.8.el7.x86_64.rpm 错误：依赖检测失败： libtirpc = 0.2.4-0.8.el7 被 libtirpc-devel-0.2.4-0.8.el7.x86_64 需要 卸载现有版本 # yum erase libtirpc-0.2.4-0.6.el7.x86_64 安装匹配版本 # yum localinstall libtirpc-0.2.4-0.8.el7.x86_64.rpm # yum localinstall libtirpc-devel-0.2.4-0.8.el7.x86_64.rpm
备注： 关闭THP方法：
“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和 “echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置， 然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置(责任编辑：IT) |