Certification-Cloud-Data-TechCPS/Create a Streaming Data Lake on Cloud Storage: Challenge Lab ARC110 at main · smdlabtech/Certification-Cloud-Data-TechCPS · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
------------------------------------------------------------------------------------------------------------------------------------------------------------------

TASK 1:

export BUCKET_NAME=

export TOPIC_ID=

export PROJECT_ID=

gcloud config set compute/region "REGION"

gcloud services disable dataflow.googleapis.com

gcloud services enable dataflow.googleapis.com

PROJECT_ID=$(gcloud config get-value project)
REGION=us-central1
AE_REGION=us-central


gsutil mb gs://$BUCKET_NAME

gcloud pubsub topics create $TOPIC_ID

gcloud app create --region=$AE_REGION


------------------------------------------------------------------------------------------------------------------------------------------------------------------


TASK 2:


gcloud scheduler jobs create pubsub publisher-job --schedule="* * * * *" \
    --topic=$TOPIC_ID --message-body="****Hello!**"


------------------------------------------------------------------------------------------------------------------------------------------------------------------

TASK 3:


gcloud scheduler jobs run publisher-job

git clone https://github.com/GoogleCloudPlatform/java-docs-samples.git
cd java-docs-samples/pubsub/streaming-analytics


mvn compile exec:java \
-Dexec.mainClass=com.examples.pubsub.streaming.PubSubToGcs \
-Dexec.cleanupDaemonThreads=false \
-Dexec.args=" \
    --project=$PROJECT_ID \
    --region=$REGION \
    --inputTopic=projects/$PROJECT_ID/topics/$TOPIC_ID \
    --output=gs://$BUCKET_NAME/samples/output \
    --runner=DataflowRunner \
    --windowSize=2"


------------------------------------------------------------------------------------------------------------------------------------------------------------------

          Congratulations, you're all done with the lab 😄                           Don't forget to subscribe our YouTube Channel😄

                                                                 Thank You!!!!
-------------------------------------------------------------------------------------------------------------------------------------------------------------------