We need to start working on the upgrade of our Hadoop clusters to version 3.x of Hadoop.
Part of this will require us to build packages for the latest stable release of Apache Bigtop, which is currently 3.3.0
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Add a thirdparty/bigtop33 component to reprepro | operations/puppet | production | +2 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T379385 Upgrade Hadoop to version 3.3.6 and Hive to version 4.0.1 | |||
| Open | BTullis | T380866 Build bigtop 3.3 packages for bullseye and bookworm |
Event Timeline
Comment Actions
Change #1098037 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Add a thirdparty/bigtop33 component to reprepro
Comment Actions
Change #1098037 merged by Btullis:
[operations/puppet@production] Add a thirdparty/bigtop33 component to reprepro
Comment Actions
I have built and packaged bigtop 3.3.0 packages for bullseye.
I used this method, from my workstation:
git clone https://github.com/apache/bigtop/ cd bigtop git checkout branch-3.3 docker run --rm -it -v `pwd`:/ws --workdir /ws bigtop/slaves:3.3.0-debian-11 . /etc/profile.d/bigtop.sh ./gradlew allclean bigtop-utils-deb bigtop-groovy-deb hadoop-deb hive-deb
This built the packages as per the bigtop branch-3.3 spec.
Then, in order to modify the packages and add a suffix with the distribution, I used the script that we wrote in our bigtop 1.5 branch.
wget https://gitlab.wikimedia.org/repos/data-engineering/bigtop/-/raw/update_bigtop_1.5_build/amend_bigtop_packages_wmf.sh chmod +x ./amend_bigtop_packages_wmf.sh ./amend_bigtop_packages_wmf.sh
Then I copied the packages to the apt server and added them to reprepro.
btullis@apt1002:~/bigtop33/deb11$ sudo -i reprepro -C thirdparty/bigtop33 includedeb bullseye-wikimedia `pwd`/*.deb Exporting indices... btullis@apt1002:~/bigtop33/deb11$ sudo -i reprepro -C thirdparty/bigtop33 list bullseye-wikimedia bullseye-wikimedia|thirdparty/bigtop33|amd64: bigtop-groovy 2.5.4-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: bigtop-utils 3.3.0-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-client 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-conf-pseudo 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-doc 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-hdfs 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-hdfs-datanode 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-hdfs-dfsrouter 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-hdfs-fuse 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-hdfs-journalnode 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-hdfs-namenode 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-hdfs-secondarynamenode 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-hdfs-zkfc 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-httpfs 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-kms 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-mapreduce 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-mapreduce-historyserver 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-yarn 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-yarn-nodemanager 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-yarn-proxyserver 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-yarn-resourcemanager 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-yarn-router 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hadoop-yarn-timelineserver 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hive 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hive-hbase 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hive-hcatalog 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hive-hcatalog-server 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hive-jdbc 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hive-metastore 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hive-server2 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hive-webhcat 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: hive-webhcat-server 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: libhdfs0 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: libhdfs0-dev 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: libhdfspp 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|amd64: libhdfspp-dev 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: bigtop-groovy 2.5.4-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: bigtop-utils 3.3.0-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hadoop-doc 3.3.6-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hive 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hive-hbase 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hive-hcatalog 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hive-hcatalog-server 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hive-jdbc 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hive-metastore 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hive-server2 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hive-webhcat 3.1.3-deb11-1 bullseye-wikimedia|thirdparty/bigtop33|i386: hive-webhcat-server 3.1.3-deb11-1
I will do the same for bookworm, although I may have to build the toolchain for this as well, since bookworm is not a supported target for bigtop 3.3.0.