> Ubuntu >

ubuntu14.04自建局域网源在线安装CDH5.6.0

  • 摘要:背景:要搭建一个可以添加节点的cloudera集群,但内部网络访问不了cloudera,能访问的主机访问速度也特别慢。实现方式:搭建一个内部的软件源,采用官网第二种方式安装1.搭建CDH源服务器源服务器地址:192.168.10.136可以连接外网下载以下包1.1.cloudera-manager相关包(下载地址:https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/pool/contrib/e/enterprise/
  • 背景:要搭建一个可以添加节点的cloudera集群,但内部网络访问不了cloudera,能访问的主机访问速度也特别慢。 
    实现方式:搭建一个内部的软件源,采用官网第二种方式安装

    1. 搭建CDH源服务器

    源服务器地址:192.168.10.136 
    可以连接外网

    下载以下包

    1.1. cloudera-manager相关包(下载地址:https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/pool/contrib/e/enterprise/)

    • cloudera-manager-agent_5.6.0-1.cm560.p0.54~trusty-cm5_amd64.deb
    • cloudera-manager-daemons_5.6.0-1.cm560.p0.54~trusty-cm5_all.deb
    • cloudera-manager-server_5.6.0-1.cm560.p0.54-trusty-cm5_all.deb

    1.2. oracle-j2sdk1.7(下载地址:https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/pool/contrib/o/)

    1.3. cdh安装包parcel(下载地址:https://archive.cloudera.com/cdh5/parcels/5.6.0/)

    • CDH-5.6.0-1.cdh5.6.0.p0.45-trusty.parcel
    • CDH-5.6.0-1.cdh5.6.0.p0.45-trusty.parcel.sha1
    • manifest.json

    1.4. 其他的依赖包(下载方式:可以用一台可以连接外网的主机,用apt-get install下载安装,下载完后在/var/cache/apt/archives里找到)

    • lsb-base psmisc bash libsasl2-modules zlib1g libsqlite3-0 libfuse2 fuse rpcbind libxslt1.1 libsasl2-modules-gssapi-mit libmysql-java python-urllib3 python-mysqldb mariadb-server-5.5
    安装源服务器

    1 创建目录/data/soft/pool //设立安装源路径

    拷贝cloudera-manager相关包、oracle-j2sdk1.7和其他的依赖包到/data/soft/pool

    2 安装dpkg-dev

    安装dpkg-dev,并执行dpkg-scanpackages 扫描依赖包并生成依赖关系gz包:

    sudo apt-get install dpkg-dev -y

    生成Packages.gz

    aliyunzixun@xxx.com:/data# dpkg-scanpackages soft/pool | gzip > soft/Packages.gz 一定要这么写,不然有错误

    3 创建目录/data/soft/cloudera,将cdh安装包parcel和manifest.json复制到/data/soft/cloudera/下

    4 安装apache2

    sudo apt-get install apache2

    建立软链接使其他主机访问

    cd /var/www/htmlln -s /data

    访问196.168.10.136/data, 可以查看到soft目录即表示apache安装配置成功

    2. 集群安装(用root用户安装)

    三台机器的ip和名字为

    192.168.10.236 hadoop-1 (内存16G) cloudera-manager-server 
    192.168.10.237 hadoop-2 (内存16G) 
    192.168.10.238 hadoop-3 (内存8G)

    2.1 配置各节点允许root ssh登陆 sudo vi /etc/ssh/sshd_config

    找到PermitRootLogin no一行,改为PermitRootLogin yes

    sudo service ssh restart 2.1 配置主节点root免密钥陆其他节点

    在hadoop-1上执行ssh-keygen -t rsa -P ”生成无密码密钥对 
    将公钥添加到认证文件中:cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys 
    将认证文件拷贝到hadoop-2和hadoop-3的/root/.ssh/目录下,使主节点免密码访问从节点

    2.2 配置/etc/hosts(各个节点都要) 192.168.10.236 hadoop-1192.168.10.237 hadoop-2192.168.10.238 hadoop-3 2.3 各节点设置安装源

    在/etc/apt/sources.list.d添加innersource.list

    vim /etc/apt/sources.list.d/innersources.list

    输入以下内容

    deb http://192.168.10.136/data soft/

    保存后刷新源(apt-get update)

    2.3 各节点安装oracle-j2sdk1.7 apt-get install oracle-j2sdk1.7update-alternatives --install /usr/bin/java java /usr/lib/jvm/java-7-oracle-cloudera/bin/java 300update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/java-7-oracle-cloudera/bin/javac 300update-alternatives --install /usr/bin/jps jps /usr/lib/jvm/java-7-oracle-cloudera/bin/jps 300 2.4 主节点(即hadoop-1)安装和配置数据库(MariaDB 5.5) apt-get install mariadb-server-5.5vim /etc/mysql/my.cnf

    下面是官方建议配置,必须改的有

    transaction-isolation = READ-COMMITTEDbinlog_format = mixed

    其他的没所谓

    [mysqld]transaction-isolation = READ-COMMITTED# Disabling symbolic-links is recommended to prevent assorted security risks;# to do so, uncomment this line:# symbolic-links = 0key_buffer = 16Mkey_buffer_size = 32Mmax_allowed_packet = 32Mthread_stack = 256Kthread_cache_size = 64query_cache_limit = 8Mquery_cache_size = 64Mquery_cache_type = 1max_connections = 550#expire_logs_days = 10#max_binlog_size = 100M#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system#and chown the specified folder to the mysql user.log_bin=/var/lib/mysql/mysql_binary_logbinlog_format = mixedread_buffer_size = 2Mread_rnd_buffer_size = 16Msort_buffer_size = 8Mjoin_buffer_size = 8M# InnoDB settingsinnodb_file_per_table = 1innodb_flush_log_at_trx_commit = 2innodb_log_buffer_size = 64Minnodb_buffer_pool_size = 4Ginnodb_thread_concurrency = 8innodb_flush_method = O_DIRECTinnodb_log_file_size = 512M[mysqld_safe]log-error=/var/log/mysqld.logpid-file=/var/run/mysqld/mysqld.pid

    重启服务service mysql restart

    创建相关数据库

    进入mysql命令行:$ mysql -u root -p进入mysql命令行后,直接复制下面的整段话并粘贴:create database amon DEFAULT CHARACTER SET utf8;grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';grant all on amon.* TO 'amon'@'CDH' IDENTIFIED BY 'amon_password';create database smon DEFAULT CHARACTER SET utf8;grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password';grant all on smon.* TO 'smon'@'CDH' IDENTIFIED BY 'smon_password';create database rman DEFAULT CHARACTER SET utf8;grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';grant all on rman.* TO 'rman'@'CDH' IDENTIFIED BY 'rman_password';create database hmon DEFAULT CHARACTER SET utf8;grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password';grant all on hmon.* TO 'hmon'@'CDH' IDENTIFIED BY 'hmon_password';create database hive DEFAULT CHARACTER SET utf8;grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';grant all on hive.* TO 'hive'@'CDH' IDENTIFIED BY 'hive_password';create database oozie DEFAULT CHARACTER SET utf8;grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie_password';grant all on oozie.* TO 'oozie'@'CDH' IDENTIFIED BY 'oozie_password';create database metastore DEFAULT CHARACTER SET utf8;grant all on metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';grant all on metastore.* TO 'hive'@'CDH' IDENTIFIED BY 'hive_password';GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'gaoying' WITH GRANT OPTION;flush privileges; 2.5 主节点安装Cloudera Manager Server apt-get install cloudera-manager-daemons cloudera-manager-server

    log目录: /var/log/cloudera-scm-server/cloudera-scm-server.log

    2.6 Cloudera Manager Server配置数据库(主节点) /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p --scm-host loclhost scm scm scm_password 2.7 启动server服务(主节点) service cloudera-scm-server start

    如果没启动成功,可查看/var/log/cloudera-scm-server/cloudera-scm-server.log 

    2.8 手动在各节点安装cloudera manager agent apt-get install lsb-base psmisc bash libsasl2-modules zlib1g libsqlite3-0 libfuse2 fuse rpcbind libxslt1.1 libsasl2-modules-gssapi-mit libmysql-java python-urllib3 python-mysqldb cloudera-manager-daemons cloudera-manager-agent 3 使用cloudera manager wizard完成安装

    ubuntu14.04自建局域网源在线安装CDH5.6.0
    在parcels的more options 
    添加Remote Parcel Repository URLS

    http://192.168.10.136/data/soft/cloudera/

    ubuntu14.04自建局域网源在线安装CDH5.6.0 
    把自建的内部源地址输入custom repository

    后续部分参照Ubuntu14.04离线安装CDH5.6.0

    3 添加节点
    1. 修改所有主机的/etc/hosts
    2. 在新主机上配置内部源,手动下载lsb-base psmisc bash libsasl2-modules zlib1g libsqlite3-0 libfuse2 fuse rpcbind libxslt1.1 libsasl2-modules-gssapi-mit libmysql-java python-urllib3 python-mysqldb cloudera-manager-daemons cloudera-manager-agent
    3. 通过cloudera manager wizard完成后续安装
  • 以上是ubuntu14.04自建局域网源在线安装CDH5.6.0的内容,更多 局域网 安装 Ubuntu 14.04 在线 5.6 cdh 的内容,请您使用右上方搜索功能获取相关信息。
(责任编辑:IT)