refactor!: Refactor Extension for Multi-Collection Support (port of #129)

POSTINSTALL.md

@@ -1,8 +1,12 @@
 ### Post-installation Notes
 
-Now that you've installed the extension, changes from `${param:FIRESTORE_COLLECTION_PATH}`
-in your Firestore Database will be synced to the Typesense Collection `${param:TYPESENSE_COLLECTION_NAME}`
-on `${param:TYPESENSE_HOSTS}`.
+Now that you've installed the extension, changes from your configured Firestore collection(s)
+will be synced to the configured Typesense collection(s) on `${param:TYPESENSE_HOSTS}`.
+
+If you installed the legacy single-collection params (`${param:FIRESTORE_COLLECTION_PATH}` and
+`${param:TYPESENSE_COLLECTION_NAME}`), the extension will still work, but it will log a deprecation
+warning. Plan to migrate to `FIRESTORE_COLLECTION_PATHS` and `TYPESENSE_COLLECTION_NAMES` in a future
+release.
 
 ### Pre-requisites
 
@@ -25,7 +29,7 @@ This will trigger the backfill background function, which will read data from yo
 
 ### See the Extension in Action
 
-Try adding or updating a Firestore document in `${param:FIRESTORE_COLLECTION_PATH}` through the Firestore UI.
+Try adding or updating a Firestore document in your configured Firestore collection through the Firestore UI.
 You should see the change reflected in your Typesense collection.
 
 ### Monitoring

PREINSTALL.md

@@ -4,6 +4,13 @@ do full-text fuzzy search on your Firestore data.
 This extension listens to your specified Firestore collection and syncs Firestore documents to Typesense 
 on creation, updates and deletes. It also provides a function to help you backfill data.
 
+This release supports both the legacy single-collection params and the newer 1-n collection params.
+The legacy single-collection params are deprecated and will emit a warning in the function logs when used.
+If both config shapes are fully set, the extension will use the newer multi-collection params and warn once.
+If only part of either config shape is set, installation will fail with a clear error.
+For new installs, use `FIRESTORE_COLLECTION_PATHS`, `TYPESENSE_COLLECTION_NAMES`,
+`FIRESTORE_COLLECTION_FIELDS_LIST`, and `FLATTEN_NESTED_DOCUMENTS_LIST`.
+
 #### Additional setup
 
 Before installing this extension, make sure that you have:

README.md

@@ -56,6 +56,15 @@ Learn more about installing extensions in the Firebase Extensions documentation:
 > [!TIP]
 > You can install this extension multiple times in your Firebase project by clicking on the installation link above multiple times, and use a different Firestore collection path in each installation instance. [Here](https://github.com/typesense/firestore-typesense-search/issues/9#issuecomment-885940705) is a screenshot of how this looks.
 
+> [!NOTE]
+> This release also supports the legacy single-collection params (`FIRESTORE_COLLECTION_PATH`,
+> `TYPESENSE_COLLECTION_NAME`, `FIRESTORE_COLLECTION_FIELDS`, and `FLATTEN_NESTED_DOCUMENTS`).
+> They are deprecated and will log a warning when used. If both config shapes are fully set, the
+> extension uses the multi-collection params and warns once. If only part of either shape is set,
+> the install fails with a clear error. For new installs, prefer the multi-collection params
+> (`FIRESTORE_COLLECTION_PATHS`, `TYPESENSE_COLLECTION_NAMES`, `FIRESTORE_COLLECTION_FIELDS_LIST`,
+> and `FLATTEN_NESTED_DOCUMENTS_LIST`).
+
 
 #### 🎛️ Configuration Parameters
 

docker-compose.yml

@@ -1,6 +1,6 @@
 services:
   typesense:
-    image: typesense/typesense:27.0
+    image: typesense/typesense:29.0
     restart: on-failure
     ports:
       - "8108:8108"

extension.yaml

@@ -34,7 +34,7 @@ resources:
           - attribute: database
             value: ${DATABASE}
           - attribute: document
-            value: ${FIRESTORE_COLLECTION_PATH}/{documentID}
+            value: "{path=**}/{documentID}"
             operator: match-path-pattern
 
   - name: backfill
@@ -101,13 +101,13 @@ params:
         value: northamerica-northeast1
       - label: northamerica-northeast2 (Toronto)
         value: northamerica-northeast2
-      
+
       # South America
       - label: southamerica-east1 (Sao Paulo)
         value: southamerica-east1
       - label: southamerica-west1 (Santiago)
         value: southamerica-west1
-      
+
       # Europe
       - label: europe-north1 (Finland)
         value: europe-north1
@@ -133,15 +133,15 @@ params:
         value: europe-central2
       - label: europe-southwest1 (Madrid)
         value: europe-southwest1
-      
+
       # Middle East
       - label: me-central1 (Doha)
         value: me-central1
       - label: me-central2 (Dammam)
         value: me-central2
       - label: me-west1 (Tel Aviv)
         value: me-west1
-      
+
       # Asia
       - label: asia-east1 (Taiwan)
         value: asia-east1
@@ -161,7 +161,7 @@ params:
         value: asia-southeast1
       - label: asia-southeast2 (Jakarta)
         value: asia-southeast2
-      
+
       # Australia
       - label: australia-southeast1 (Sydney)
         value: australia-southeast1
@@ -288,19 +288,94 @@ params:
   - param: FIRESTORE_COLLECTION_PATH
     label: Firestore Collection Path
     description: >-
-      The Firestore collection that needs to be indexed into Typesense.
+      Legacy single-collection configuration. Use this only if you are syncing one Firestore collection.
+      For new installs, prefer FIRESTORE_COLLECTION_PATHS.
     example: path/to/firestore_collection
     validationRegex: "^[^/]+(/[^/]+/[^/]+)*$"
     validationErrorMessage: Firestore collection paths must be an odd number of segments separated by slashes, e.g. "path/to/firestore_collection".
-    default: path/to/firestore_collection
-    required: true
+    default: ""
+    required: false
+
   - param: FIRESTORE_COLLECTION_FIELDS
     label: Firestore Collection Fields
     description: >-
-      A comma separated list of fields that need to be indexed from each Firestore document. Leave blank to index all fields.
+      Legacy comma-separated list of fields to index from a single Firestore collection.
+      Leave blank to index all fields. For new installs, prefer FIRESTORE_COLLECTION_FIELDS_LIST.
     example: field1,field2,field3
     default: ""
     required: false
+
+  - param: TYPESENSE_COLLECTION_NAME
+    label: Typesense Collection Name
+    description: >-
+      Legacy Typesense collection name for a single Firestore collection.
+      For new installs, prefer TYPESENSE_COLLECTION_NAMES.
+    default: ""
+    required: false
+
+  - param: FLATTEN_NESTED_DOCUMENTS
+    label: Flatten Nested Documents
+    description: >-
+      Legacy flatten setting for a single Firestore collection.
+      Set to "Yes" for Typesense versions 0.23.1 and earlier.
+      For new installs, prefer FLATTEN_NESTED_DOCUMENTS_LIST.
+    type: select
+    options:
+      - label: No
+        value: false
+      - label: Yes
+        value: true
+    required: false
+
+  - param: FIRESTORE_COLLECTION_PATHS
+    label: Firestore Collection Paths
+    description: >-
+      Comma-separated list of Firestore collection paths to index into Typesense.
+      Supports both regular collections and subcollections with path parameters (e.g., "users/{userId}/books").
+      For existing single-collection installs, this parameter can be left blank.
+    example: users,products,users/{userId}/books,stores/{storeId}/products
+    validationRegex: "^[^,]+(?:,[^,]+)*$"
+    validationErrorMessage: Collection paths must be comma-separated without spaces.
+    default: ""
+    required: false
+
+  - param: TYPESENSE_COLLECTION_NAMES
+    label: Typesense Collection Names
+    description: >-
+      Comma-separated list of Typesense collection names corresponding to the Firestore collection paths.
+      The order must match the FIRESTORE_COLLECTION_PATHS parameter.
+      For existing single-collection installs, this parameter can be left blank.
+    example: users,products,user_books,store_products
+    validationRegex: "^[^,]+(?:,[^,]+)*$"
+    validationErrorMessage: Collection names must be comma-separated without spaces.
+    default: ""
+    required: false
+
+  - param: FIRESTORE_COLLECTION_FIELDS_LIST
+    label: Firestore Collection Fields List
+    description: >-
+      Pipe-separated list of comma-separated field lists for each collection.
+      Each field list corresponds to a collection in FIRESTORE_COLLECTION_PATHS.
+      Leave empty or use "||" to skip fields for a collection.
+    example: name,email,profile|title,description,price||name,email|title,description
+    default: ""
+    required: false
+
+  - param: FLATTEN_NESTED_DOCUMENTS_LIST
+    label: Flatten Nested Documents List
+    description: >-
+      Comma-separated list of boolean values indicating whether to flatten nested documents for each collection.
+      Each value corresponds to a collection in FIRESTORE_COLLECTION_PATHS.
+    type: select
+    options:
+      - label: No
+        value: false
+      - label: Yes
+        value: true
+    example: false,true,false,true
+    default: false,true
+    required: false
+
   - param: TYPESENSE_HOSTS
     label: Typesense Hosts
     description: >-
@@ -317,25 +392,7 @@ params:
       Click on "Generate API Key" in cluster dashboard in Typesense Cloud
     example: ""
     required: true
-  - param: TYPESENSE_COLLECTION_NAME
-    label: Typesense Collection Name
-    description: >-
-      Typesense collection name to index data into. This collection needs to exist before this extension is installed. Please create it via the Typesense Cloud dashboard or API.
-    default: companies
-    required: true
-  - param: FLATTEN_NESTED_DOCUMENTS
-    label: Flatten Nested Documents
-    description: >-
-      Should nested documents in Firestore be flattened by this extension before they are indexed in Typesense? 
-      Set to "Yes" for Typesense versions 0.23.1 and earlier. Set to "No" for Typesense versions 0.24.0 and later.
-    type: select
-    options:
-      - label: No
-        value: false
-      - label: Yes
-        value: true
-    default: false
-    required: false
+
   - param: LOG_TYPESENSE_INSERTS
     label: Log Typesense Inserts for Debugging
     description: >-

extensions/test-params-field-filtering-subcollections.local.env

@@ -0,0 +1,11 @@
+LOCATION=us-central1
+FIRESTORE_DATABASE_REGION=nam5
+FIRESTORE_COLLECTION_PATHS=test_users/{userId}/test_books,test_stores/{storeId}/test_products
+TYPESENSE_COLLECTION_NAMES=test_user_books,test_store_products
+FIRESTORE_COLLECTION_FIELDS_LIST=title,author,rating|name,price,category
+FLATTEN_NESTED_DOCUMENTS_LIST=false,true
+TYPESENSE_HOSTS=localhost
+TYPESENSE_PORT=8108
+TYPESENSE_PROTOCOL=http
+TYPESENSE_API_KEY=xyz
+LOG_TYPESENSE_INSERTS=true

extensions/test-params-field-filtering.local.env

@@ -0,0 +1,11 @@
+LOCATION=us-central1
+FIRESTORE_DATABASE_REGION=nam5
+FIRESTORE_COLLECTION_PATHS=test_users,test_products
+TYPESENSE_COLLECTION_NAMES=test_users,test_products
+FIRESTORE_COLLECTION_FIELDS_LIST=name,email,profile|title,description,nested_field
+FLATTEN_NESTED_DOCUMENTS_LIST=false,true
+TYPESENSE_HOSTS=localhost
+TYPESENSE_PORT=8108
+TYPESENSE_PROTOCOL=http
+TYPESENSE_API_KEY=xyz
+LOG_TYPESENSE_INSERTS=true

extensions/test-params-multi-collection-subcollections.local.env

@@ -0,0 +1,11 @@
+LOCATION=us-central1
+FIRESTORE_DATABASE_REGION=nam5
+FIRESTORE_COLLECTION_PATHS=users/{userId}/books,stores/{storeId}/products
+TYPESENSE_COLLECTION_NAMES=user_books,store_products
+FIRESTORE_COLLECTION_FIELDS_LIST=title,author,rating|name,price,category
+FLATTEN_NESTED_DOCUMENTS_LIST=false,true
+TYPESENSE_HOSTS=localhost
+TYPESENSE_PORT=8108
+TYPESENSE_PROTOCOL=http
+TYPESENSE_API_KEY=xyz
+LOG_TYPESENSE_INSERTS=true

extensions/test-params-multi-collection.local.env

@@ -0,0 +1,11 @@
+LOCATION=us-central1
+FIRESTORE_DATABASE_REGION=nam5
+FIRESTORE_COLLECTION_PATHS=users,products,customers,orders
+TYPESENSE_COLLECTION_NAMES=users,products,customers,orders
+FIRESTORE_COLLECTION_FIELDS_LIST=||name,email,profile|title,description,nested_field
+FLATTEN_NESTED_DOCUMENTS_LIST=false,true,false,true
+TYPESENSE_HOSTS=localhost
+TYPESENSE_PORT=8108
+TYPESENSE_PROTOCOL=http
+TYPESENSE_API_KEY=xyz
+LOG_TYPESENSE_INSERTS=true