-
Notifications
You must be signed in to change notification settings - Fork 241
feat: add integration test cases for 2pc #865
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 30 commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
f7ee305
chore: add integration test cases for 2pc
vaibhav-datazip 652d926
fix: logical issue in cdc check condition
vaibhav-datazip e461a6a
Merge branch 'staging' into feat/2pc-integration-tests
vaibhav-datazip 6ee9bfd
Merge branch 'staging' into feat/2pc-integration-tests
vaibhav-datazip 1fd2f83
chore: resolve merge conflict
vaibhav-datazip 64658bc
fix: improve testcases
vaibhav-datazip 85f7f4f
fix: skip postgres lsn check when all streams are finished
vaibhav-datazip 631689e
Merge branch 'staging' into feat/2pc-integration-tests
vaibhav-datazip 80f0ccc
chore: increase integration test timeout
vaibhav-datazip ccf2f90
fix: mssql incremental integration test
vaibhav-datazip 54c2bde
chore: replace unnecessary blank sync case as pre setup command
vaibhav-datazip a387c3a
fix: mongo recovery sync behaviour
vaibhav-datazip 865afc6
Merge branch 'staging' into feat/2pc-integration-tests
hash-data 2061855
fix: add comments for test cases
vaibhav-datazip 5c77aa7
fix: trivy vulnerability error for pgx
vaibhav-datazip e9a7b35
Merge branch 'staging' into feat/2pc-integration-tests
vaibhav-datazip 548da83
Merge branch 'staging' into feat/2pc-integration-tests
vaibhav-datazip c6ccaf8
chore: separate 2pc cases workflow
vaibhav-datazip 4f1cd09
chore: resolve merge conflict
vaibhav-datazip 663a3e9
fix: non utf 8 fields for insert 2pc
vaibhav-datazip 44828a1
Merge branch 'staging' into feat/2pc-integration-tests
vaibhav-datazip e07688f
Merge branch 'staging' into feat/2pc-integration-tests
vaibhav-datazip 6e6fe08
fix: remove schema validation during 2pc tests
vaibhav-datazip 8706e7f
fix: bump up toolchain versions for 2pc tests
vaibhav-datazip 4b2ea89
fix: remove Integration from 2pc test to avoid incorrect string matching
vaibhav-datazip 53a0ab6
fix: 2pc table name
vaibhav-datazip 05af31d
Merge branch 'staging' into feat/2pc-integration-tests
vaibhav-datazip 49bd1c3
chore: removal of unused code
vaibhav-datazip 41d846e
chore: abstract 2pc and integration test
vaibhav-datazip 9f0c6e4
fix: run tests separately abstract out the code
vaibhav-datazip 465379e
chore: add log for recovery lsn validation skip
vaibhav-datazip File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| name: 2PC Integration Tests | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - "master" | ||
| paths: | ||
| - '**/*.go' | ||
| - '**/*.java' | ||
| pull_request: | ||
| branches: | ||
| - "*" | ||
| paths: | ||
| - '**/*.go' | ||
| - '**/*.java' | ||
|
|
||
| jobs: | ||
| twopc-integration-tests: | ||
| uses: ./.github/workflows/integration-tests-runner.yml | ||
| with: | ||
| test_type: 2pc | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| name: Integration Test Runner | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| test_type: | ||
| description: 'Type of tests to run: integration or 2pc' | ||
| required: true | ||
| type: string | ||
|
|
||
| env: | ||
| MYSQL_ROOT_PASSWORD: root1234 | ||
|
|
||
| jobs: | ||
| run-tests: | ||
| name: ${{ inputs.test_type == '2pc' && '2PC Integration Tests' || 'Integration Tests' }} | ||
| environment: integration_tests | ||
| runs-on: 32gb-runner | ||
| timeout-minutes: 45 | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v3 | ||
|
|
||
| - name: Set up Go | ||
| uses: actions/setup-go@v4 | ||
| with: | ||
| go-version-file: "go.mod" | ||
|
|
||
| - name: Set up Java for Maven | ||
| uses: actions/setup-java@v3 | ||
| with: | ||
| distribution: 'temurin' | ||
| java-version: '17' | ||
|
|
||
| - name: Install DB2 CLI Driver | ||
| run: | | ||
| go run github.com/ibmdb/go_ibm_db/installer@v0.5.4 | ||
| cp -r "$(go env GOPATH)/pkg/mod/github.com/ibmdb/clidriver" ./clidriver | ||
| IBM_DB_HOME="$(pwd)/clidriver" | ||
| echo "IBM_DB_HOME=$IBM_DB_HOME" >> $GITHUB_ENV | ||
| echo "CGO_CFLAGS=-I$IBM_DB_HOME/include" >> $GITHUB_ENV | ||
| echo "CGO_LDFLAGS=-L$IBM_DB_HOME/lib -Wl,-rpath,$IBM_DB_HOME/lib" >> $GITHUB_ENV | ||
| echo "LD_LIBRARY_PATH=$IBM_DB_HOME/lib" >> $GITHUB_ENV | ||
|
|
||
| - name: Start Test Infrastructure | ||
| run: | | ||
| docker compose -f ./drivers/mysql/docker-compose.yml up -d | ||
| docker compose -f ./drivers/postgres/docker-compose.yml up -d | ||
| docker compose -f ./drivers/mongodb/docker-compose.yml up -d | ||
| docker compose -f ./drivers/oracle/docker-compose.yml up -d | ||
| docker compose -f ./drivers/db2/docker-compose.yml up -d | ||
| docker compose -f ./drivers/mssql/docker-compose.yml up -d | ||
| docker compose -f ./destination/iceberg/local-test/docker-compose.yml up minio mc postgres spark-iceberg -d | ||
|
|
||
| - name: Start Kafka | ||
| if: inputs.test_type == 'integration' | ||
| run: docker compose -f ./drivers/kafka/docker-compose.yml up -d | ||
|
|
||
| - name: Wait for MySQL | ||
| uses: nick-fields/retry@v2 | ||
| with: | ||
| timeout_minutes: 5 | ||
| max_attempts: 30 | ||
| retry_wait_seconds: 5 | ||
| command: | | ||
| docker exec olake_mysql-test mysql -h localhost -u root -p${{ env.MYSQL_ROOT_PASSWORD }} -e "SELECT 1" | ||
|
|
||
| - name: Wait for PostgreSQL | ||
| uses: nick-fields/retry@v2 | ||
| with: | ||
| timeout_minutes: 5 | ||
| max_attempts: 30 | ||
| retry_wait_seconds: 5 | ||
| command: | | ||
| docker exec olake_postgres-test psql -h localhost -U postgres -d postgres -c "SELECT 1" | ||
|
|
||
| - name: Wait for MongoDB | ||
| uses: nick-fields/retry@v2 | ||
| with: | ||
| timeout_minutes: 5 | ||
| max_attempts: 30 | ||
| retry_wait_seconds: 5 | ||
| command: | | ||
| docker exec primary_mongo mongosh --host localhost --port 27017 -u mongodb -p secure_password123 --authenticationDatabase admin --eval "db.adminCommand('ping')" | ||
|
|
||
| - name: Wait for Oracle | ||
| uses: nick-fields/retry@v2 | ||
| with: | ||
| timeout_minutes: 5 | ||
| max_attempts: 30 | ||
| retry_wait_seconds: 5 | ||
| command: | | ||
| docker exec oracle-23c bash -c "echo 'SELECT 1 FROM dual;' | sqlplus -s system/secret1234@//localhost:1521/ORCL" | ||
|
|
||
| - name: Wait for DB2 | ||
| uses: nick-fields/retry@v2 | ||
| with: | ||
| timeout_minutes: 10 | ||
| max_attempts: 30 | ||
| retry_wait_seconds: 25 | ||
| command: | | ||
| docker exec db2-test bash -c "su - db2inst1 -c 'db2 connect to TESTDB'" | ||
|
|
||
| - name: Wait for Kafka | ||
| if: inputs.test_type == 'integration' | ||
| uses: nick-fields/retry@v2 | ||
| with: | ||
| timeout_minutes: 5 | ||
| max_attempts: 30 | ||
| retry_wait_seconds: 5 | ||
| command: | | ||
| docker exec kafkaJson kafka-topics --bootstrap-server localhost:9092 --list | ||
| docker exec kafkaAvro kafka-topics --bootstrap-server localhost:9092 --list | ||
| curl -f http://localhost:8081/subjects | ||
|
|
||
| - name: Wait for MSSQL | ||
| uses: nick-fields/retry@v2 | ||
| with: | ||
| timeout_minutes: 5 | ||
| max_attempts: 30 | ||
| retry_wait_seconds: 5 | ||
| command: | | ||
| docker exec olake-mssql /opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P 'Password!123' -C -Q "SELECT 1" | ||
|
|
||
| - name: Initialize MSSQL Database | ||
| run: | | ||
| docker exec olake-mssql /opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P 'Password!123' -C -d master -i /docker-entrypoint-initdb.d/01-init.sql | ||
|
|
||
| - name: Set up Data Directories | ||
| run: | | ||
| sudo mkdir -p /home/runner/work/olake/olake/destination/iceberg/local-test/data/postgres-data | ||
| sudo mkdir -p /home/runner/work/olake/olake/destination/iceberg/local-test/data/minio-data | ||
| sudo mkdir -p /home/runner/work/olake/olake/destination/iceberg/local-test/data/ivy-cache | ||
| sudo chown -R 999:999 /home/runner/work/olake/olake/destination/iceberg/local-test/data | ||
| sudo chmod -R 777 /home/runner/work/olake/olake/destination/iceberg/local-test/data | ||
|
|
||
| - name: Install Go Dependencies | ||
| run: go mod download | ||
|
|
||
| - name: Build Project | ||
| run: go build -v ./... | ||
|
|
||
| - name: Cache Maven dependencies | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: | | ||
| ~/.m2/repository | ||
| key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-maven- | ||
|
|
||
| - name: Build Iceberg Sink | ||
| working-directory: ./destination/iceberg/olake-iceberg-java-writer | ||
| run: mvn clean package -DskipTests | ||
|
|
||
| - name: Run Integration Tests | ||
| if: inputs.test_type == 'integration' | ||
| run: | | ||
| go test -v -p 7 ./drivers/kafka/internal/... ./drivers/mysql/internal/... ./drivers/postgres/internal/... ./drivers/mongodb/internal/... ./drivers/oracle/internal/... ./drivers/db2/internal/... ./drivers/mssql/internal/... -timeout 0 -run 'Integration' | ||
|
|
||
| - name: Run 2PC Integration Tests | ||
| if: inputs.test_type == '2pc' | ||
| run: | | ||
| go test -v -p 6 ./drivers/mysql/internal/... ./drivers/postgres/internal/... ./drivers/mongodb/internal/... ./drivers/mssql/internal/... ./drivers/oracle/internal/... ./drivers/db2/internal/... -timeout 0 -run '2PC' | ||
|
|
||
| - name: Cleanup | ||
| if: always() | ||
| run: | | ||
| docker compose -f ./destination/iceberg/local-test/docker-compose.yml down | ||
| docker compose -f ./drivers/mysql/docker-compose.yml down | ||
| docker compose -f ./drivers/postgres/docker-compose.yml down | ||
| docker compose -f ./drivers/oracle/docker-compose.yml down | ||
| docker compose -f ./drivers/mongodb/docker-compose.yml down | ||
| docker compose -f ./drivers/db2/docker-compose.yml down | ||
| docker compose -f ./drivers/mssql/docker-compose.yml down | ||
|
|
||
| - name: Cleanup Kafka | ||
| if: always() && inputs.test_type == 'integration' | ||
| run: docker compose -f ./drivers/kafka/docker-compose.yml down |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.