Basic Hibuz Bigdata Stack information

The Hibuz Bigdata Stack (HBS) is a fully containerized, production-ready big data development environment tailored for modern data engineering and analytics. Built on top of Ubuntu 26.04 LTS and Java 21, it delivers a pre-configured, localized ecosystem that allows developers to spin up a comprehensive big data stack instantly using Docker Compose.

Components

• Orchestration and notebooks: Apache Zeppelin
• Data warehousing: Apache Hive
• Processing: Apache Spark and Apache Flink
• Storage and table formats: Apache Hadoop (HDFS), Apache HBase, and Apache Iceberg

Version values and release metadata are intentionally maintained in .agents/context/versions.env, not duplicated in this overview.

Quick usage for hadoop-dev docker image

• Docker build and run

git clone https://github.com/hibuz/hadoop-docker
cd hadoop-docker

docker compose up hadoop-dev --no-build

Docker build & run for custom hadoop user and version

• see Dockerfile

Hadoop Docker Build & Push Order

# hadoop
hadoop-docker$ docker build -t hibuz/hadoop-dev .
# hbase|spark|hive|flink
hadoop-docker/(hbase|spark|hive|flink|zepplein)$ docker compose up --build
# flink-base for zeppelin
hadoop-docker/zeppelin$ docker compose build flink-base
# zeppelin
hadoop-docker/zeppelin$ docker compose up --build


# docker taagging & push
docker tag hibuz/hadoop-dev hibuz/hadoop-dev:3.x.x
docker push hibuz/hadoop-dev
docker push hibuz/hadoop-dev:3.x.x

Attach to running container

docker exec -it hadoop bash

Prepare input files into the distributed filesystem

# Make the HDFS directories
hdfs dfs -mkdir -p /user/hadoop/input
# Copy the input files
hdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml input

Run some of the examples provided:

# Run example wordcount job:
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount input output
# View the output files on the distributed filesystem:
hdfs dfs -cat output/*

# Run example wordcount grep job:
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep input output/count 'dfs[a-z.]+'
# View the output files on the distributed filesystem:
hdfs dfs -cat output/count/*
# Result of the output files 
1	dfsadmin
1	dfs.replication

# Remove the output dir:
hdfs dfs -rm -r output

Visit hadoop dashboard

• Hadoop Dashboard: http://localhost:9870
• Yarn Dashboard: http://localhost:8088 (run start-yarn.sh or uncomment command props in docker-compose.yml)
• Hadoop Job History: http://localhost:19888

Stops containers and removes containers, networks, and volumes created by compose up.

docker compose down -v

[+] Running 3/3
 ✔ Container hbase         Removed
 ✔ Volume hbase_hbase-vol  Removed
 ✔ Network hbase_default   Removed

Reference

• Execute MapReduce jobs
• https://github.com/rancavil/hadoop-single-node-cluster
• https://github.com/big-data-europe/docker-hadoop

TODOs

Flink 2.2.1 error
 Flink SQL> CREATE CATALOG myhive WITH (
    >     'type' = 'hive',
   [ERROR] Could not execute SQL statement. Reason: org.apache.flink.table.api.ValidationException: Could not find any factory for identifier 'hive' that implements 'org.apache.flink.table.factories.CatalogFactory' in the classpath.