Merge tag 'REL_16_9' into Cloudberry

Merge upstream PostgreSQL REL_16_9 into Cloudberry, upgrading the
kernel from PG 14.4 to PG 16.9. This spans two major versions (PG 15
and PG 16), bringing in 5730 commits.

Key features and changes, sorted by importance:

1. MERGE command
   - SQL-standard MERGE for INSERT/UPDATE/DELETE in a single statement.
   - ruleutils decompilation, PlaceHolderVar support in actions
   - Fixes for partitioned tables, cross-partition triggers, RLS,
   - EvalPlanQual, self-modified tuples, pg_stat_statements

2. Logical replication enhancements
   - Built-in support for prepared transactions (2PC)
   - Enable two-phase via replication protocol
   - Streaming changes after speculative aborts
   - New pg_create_subscription predefined role
   - Parallel apply workers (PG 16)
   - Row/column filtering, schema-level publication (PG 15)
   - Critical fix for data loss in logical replication
   - Slot invalidation, snapshot, and memory leak fixes

3. SQL/JSON
   - IS JSON predicate
   - Standard JSON constructor functions (json_array, json_object,
     json_scalar, json_serialize)
   - Auto-generated query jumbling via gen_node_support.pl
   - Fix json_array() subquery double transformation

4. WAL and storage subsystem
   - LZ4 compression for full-page writes (FPW)
   - Custom WAL resource managers
   - Skip WAL recycling/preallocation during archive recovery
   - XLOG_FPI_FOR_HINT honors full_page_writes setting
   - Direct I/O support on macOS
   - Buffer manager infrastructure refactor for faster relation extension
   - New smgrzeroextend(), FileZero(), FileFallocate()
   - Remove HeapBitmapScan skip_fetch optimization (correctness issue)
   - WAL/Recovery separation: xlogrecovery.c, xlogprefetcher.c (PG 15)
   - Backup sink architecture (PG 15)

5. Query optimizer / executor
   Performance:
   - Hash table acceleration for NOT IN(values)
   - Datum sorts for single-column sorts
   - Result Cache renamed to Memoize with ongoing fixes
   - Parallel Hash Full Join
   - Reduced planning cost for deeply-nested views
   - Fewer pallocs when building partition bounds
   - Outer join optimization enhancement (PG 16)
   - Symmetric hash join optimization (PG 16)
   Critical fixes:
   - Planner failure to identify multiple hashable ScalarArrayOpExprs
   - Consistent whole-row Var construction in parsing and planning
   - WindowAgg evaluation crash
   - WITH RECURSIVE UNION assert failure
   - ORDER BY / DISTINCT aggregates with FILTER
   - setrefs.c missing expression processing on prune steps
   - Parallel Hash Join extreme skew detection

6. pgstat monitoring enhancements
   pgstat shared memory rewrite (PG 15):
   - Stats collector process removed; stats now in shared memory
   - Monolithic pgstat.c split into 14 files under utils/activity/
   pg_stat_io (PG 16 new view):
   - Detailed I/O statistics: shared buffer hits, I/O times, writeback
   pg_stat_statements:
   - JIT counters, temp file block I/O timing
   - Utility query string normalization, MERGE support
   - 32-bit integer overflow fix
   Other:
   - pg_stat_wal time accumulation as instr_time
   - Macro-generated pg_stat_get*() functions for tables and databases
   - Enhanced pg_stat_reset_single_table_counters
   - SP-GiST index scans counted in pg_stat
   - New test helpers: pg_stat_force_next_flush(), pg_stat_have_stats()

7. libpq and client protocol
   - New PQsendFlushRequest
   - Pipeline mode state machine fix
   - SASL code refactored to generic interface
   - Escape function fixes for invalid encoding data (security)
   - PQescapeLiteral()/PQescapeIdentifier() length handling fix
   - New pg_encoding_set_invalid()
   - Build-time check that libpq doesn't call exit()/abort()

8. Security and privilege model
   - CREATEROLE privilege restrictions (major security improvement)
   - New GUCs: createrole_self_grant, reserved_connections
   - New predefined roles: pg_maintain, pg_create_subscription
   - Non-superuser predefined roles for vacuum/analyze
   - Revoke PUBLIC CREATE from public schema (now owned by pg_database_owner)
   - Security invoker views (SECURITY INVOKER)
   - session_authorization and role interaction fixes
   - scram_SaltedPassword() integer overflow fix

9. pg_dump / pg_upgrade / pg_basebackup toolchain
    pg_dump:
    - Generic compression API, zstd support, LZ4 frame-only format
    pg_basebackup:
    - Extended compression options, server-side compression with -Fp
    - Client-side LZ4 decompression, parallel zstd compression
    pg_upgrade:
    - Fix unintentional 'NULL' string literal
    - Fix XMLSERIALIZE(NO INDENT) cross-version upgrade
    - pg_dumpall handling of dangling OIDs in pg_auth_members

10. Partitioned table improvements
    - Self-referencing FKs in partitioned tables
    - Detach partition with top-level FK fix
    - Reset relhassubclass on ATTACH
    - Trigger rename consistency, preserve firing state on clone
    - Disallow partitionwise join/grouping on collation mismatch

11. ICU and collation
    - Build ICU support by default
    - initdb uses uloc_getDefault()
    - CREATE DATABASE LOCALE applies to all collation providers

12. Node support function auto-generation (FIXME)
    - gen_node_support.pl auto-generates copyfuncs/equalfuncs/outfuncs/readfuncs
    - Query jumbling code auto-generation
    - NodeTag ABI stability check

13. Global renames and structural changes
    - RelFileNode -> RelFileLocator (pervasive rename across hundreds of files)
    - GUC system split: guc.c -> guc.c + guc_tables.c + guc_funcs.c
    - varatt.h separated from postgres.h
    - Test framework: PostgresNode.pm -> PostgreSQL::Test::Cluster

14. SQL types and functions
    - Numeric scale allows negative or greater than precision
    - New ANY_VALUE aggregate function
    - unnest(multirange), range_agg with multirange inputs
    - pg_size_pretty/pg_size_bytes support petabytes
    - ALTER TABLE SET ACCESS METHOD
    - SYSTEM_USER function
    - numeric_mul() overflow fix

15. psql improvements
    - PSQL_WATCH_PAGER for \watch command
    - New \drg command for role grants
    - \copy from sends data in larger chunks
    - pg_waldump --rmgr multiple specification

16. Critical data corruption and crash fixes
    - Fix data corruption when relation truncation fails
    - Fail instead of corrupting page header on non-LP_NORMAL TID update
    - Fix unsafe BufferDescriptors access
    - GB18030 SIGSEGV from out-of-bounds read
    - Snowball stemmer null pointer dereference after OOM
    - Rare standby assertion failure on primary restart
    - catcache invalidation during list entry construction

17. Vacuum / Autovacuum
    - Parallel VACUUM (PG 16, vacuumparallel.c new file)
    - ON COMMIT DELETE ROWS avoids ERROR after relhassubclass=f
    - Use WaitLatch() instead of pg_usleep() at end-of-vacuum truncation
    - Prevent numeric overflow in parallel numeric aggregates
    - ANALYZE preserves relhasindex for partitioned tables

Co-authored-by: liushengsong <lss602726449@gmail.com>
Co-authored-by: reshke <reshke@double.cloud>
Co-authored-by: Hao Wu <gfphoenix78@gmail.com>
Co-authored-by: Jianghua.yjh <yjhjstz@gmail.com>
Co-authored-by: Dianjin Wang <wangdianjin@gmail.com>

Author: 6 people

.asf.yaml

@@ -97,7 +97,8 @@ github:
           - pax-ic-isolation2-opt-on
           - ic-expandshrink
           - ic-singlenode
-          - ic-resgroup-v2
+          # MERGE16_FIXME: enable it later
+          # - ic-resgroup-v2
           - ic-contrib
           - ic-gpcontrib
           - ic-fixme

.git-blame-ignore-revs

@@ -12,7 +12,37 @@
 #
 # Add new entries by adding the output of the following to the top of the file:
 #
-# $ git log --pretty=format:"%H # %cd %n# %s" $PGINDENTGITHASH -1
+# $ git log --pretty=format:"%H # %cd%n# %s" $PGINDENTGITHASH -1 --date=iso
+
+b334612b8aee9f9a34378982d8938b201dfad323 # 2023-06-20 09:50:43 -0400
+# Pre-beta2 mechanical code beautification.
+
+0245f8db36f375326c2bae0c3420d3c77714e72d # 2023-05-19 17:24:48 -0400
+# Pre-beta mechanical code beautification.
+
+b6dfee28f2b44e28b123b77a91fb05c47da63501 # 2023-03-09 15:09:45 +0900
+# Run pgindent on libpq's fe-auth.c, fe-auth-scram.c and fe-connect.c
+
+250c8ee07ede8a104ae1471f6ebca204d94267dd # 2023-01-13 15:23:17 -0800
+# Manual cleanup and pgindent of pgstat and bufmgr related code
+
+b2e6e768230be334b12dae536ba4c147fba4e9c9 # 2022-09-08 14:01:13 +0700
+# Run perltidy over Catalog.pm
+
+82d0ffae3219e4bc153a1306ce23013d168e04a2 # 2022-06-30 11:03:03 -0400
+# pgindent run prior to branching v15.
+
+23e7b38bfe396f919fdb66057174d29e17086418 # 2022-05-12 15:17:30 -0400
+# Pre-beta mechanical code beautification.
+
+a3a75b982b5bb6fba95ad8b3d48e70439dcd2329 # 2022-03-19 11:42:22 -0700
+# pgstat: run pgindent on pgstat.c/h.
+
+d231be00cbf29bf24e5b4fc79e587982bbc00ebb # 2022-02-16 09:22:34 +0200
+# Run pgindent on xlog.c.
+
+e1c1c30f635390b6a3ae4993e8cac213a33e6e3f # 2021-06-28 11:05:54 -0400
+# Pre branch pgindent / pgperltidy run
 
 # This file contains a list of commits that are unlikely what you are looking
 # for in a typical git-blame, such as mass reformatting or renaming. There are
@@ -42,6 +72,9 @@
 def5b065ff22a16a80084587613599fe15627213 # 2021-05-12 13:14:10 -0400
 # Initial pgindent and pgperltidy run for v14.
 
+ed43677e20369040ca4e50c698010c39d5ac0f47 # 2021-01-19 08:10:13 +0530
+# pgindent worker.c.
+
 8b411b8ff41566a1aa601d1f05aeebbebbdb4a54 # 2021-01-13 16:14:38 -0500
 # Run reformat-dat-files to declutter the catalog data files.
 
@@ -54,7 +87,7 @@ fa27dd40d5c5f56a1ee837a75c97549e992e32a4 # 2020-05-16 11:54:51 -0400
 e02ad575d8ab6b44500d2a3fd8c3212345e3aa2b # 2020-05-16 11:49:14 -0400
 # Final pgindent run with pg_bsd_indent version 2.1.
 
-5cbfce562f7cd2aab0cdc4694ce298ec3567930e # 2020-05-14 13:06:38 -0400
+5cbfce562f7cd2aab0cdc4694ce298ec3567930e # 2020-05-14 13:06:50 -0400
 # Initial pgindent and pgperltidy run for v13.
 
 b78542b9e975494bba6db2d2802439c3328d5ef7 # 2020-02-15 14:58:30 -0500
@@ -96,7 +129,7 @@ eaedf0df7197b21182f6c341a44e4fdaa3cd6ea6 # 2017-11-29 09:24:24 -0500
 382ceffdf7f620d8f2d50e451b4167d291ae2348 # 2017-06-21 15:35:54 -0400
 # Phase 3 of pgindent updates.
 
-c7b8998ebbf310a156aa38022555a24d98fdbfb4 # 2017-06-21 15:18:54 -0400
+c7b8998ebbf310a156aa38022555a24d98fdbfb4 # 2017-06-21 15:19:25 -0400
 # Phase 2 of pgindent updates.
 
 e3860ffa4dd0dad0dd9eea4be9cc1412373a8c89 # 2017-06-21 14:39:04 -0400
@@ -126,16 +159,25 @@ b5bce6c1ec6061c8a4f730d927e162db7e2ce365 # 2016-08-15 13:42:51 -0400
 de94e2af184e25576b13cbda8cf825118835d1cd # 2016-04-06 11:34:02 -0400
 # Run pgindent on a batch of (mostly-planner-related) source files.
 
+be060cbcd42737693f6fd425db4c139121181cce # 2016-03-09 13:51:11 -0500
+# Re-pgindent vacuumlazy.c.
+
+0231f838565d2921a0960407c4240237ba1d56ae # 2016-02-08 15:17:40 -0500
+# Re-pgindent varlena.c.
+
 d0cd7bda97a626049aa7d247374909c52399c413 # 2016-02-04 22:30:08 -0500
 # postgres_fdw: pgindent run.
 
-befa3e648ce018d84cd2a0df701927c56fe3da4e # 2015-05-24 21:44:57 -0400
+8d290c8ec6c182a4df1d089c21fe84c7912f01fe # 2016-01-17 19:13:18 -0500
+# Re-pgindent a few files.
+
+befa3e648ce018d84cd2a0df701927c56fe3da4e # 2015-05-24 21:45:01 -0400
 # Revert 9.5 pgindent changes to atomics directory files
 
 807b9e0dff663c5da875af7907a5106c0ff90673 # 2015-05-23 21:35:49 -0400
 # pgindent run for 9.5
 
-785941cdc359c6e595201ffb0df9d28f3f7173a4 # 2015-03-26 14:03:19 -0400
+785941cdc359c6e595201ffb0df9d28f3f7173a4 # 2015-03-26 14:03:25 -0400
 # Tweak __attribute__-wrapping macros for better pgindent results.
 
 7584649a1c58029a83a7a57d74cedcf1af434c97 # 2014-10-17 12:19:05 -0400
@@ -183,6 +225,9 @@ f99a569a2ee3763b4ae174e81250c95ca0fdcbb6 # 2006-10-04 00:30:14 +0000
 1dc34982511d91ef8a2b71bdcb870f067c1b3da9 # 2005-10-15 02:49:52 +0000
 # Standard pgindent run for 8.1.
 
+b3364fc81b78538006da3b3ad5680530d2af8c47 # 2005-09-22 20:44:36 +0000
+# pgindent new GIST index code, per request from Tom.
+
 15d3f9f6b7849a70281f151f0def7a6d335767d7 # 2004-08-30 02:54:42 +0000
 # Another pgindent run with lib typedefs added.
 

.gitattributes

@@ -1,8 +1,9 @@
 *		whitespace=space-before-tab,trailing-space
 *.[chly]	whitespace=space-before-tab,trailing-space,indent-with-non-tab,tabwidth=4
 *.pl		whitespace=space-before-tab,trailing-space,tabwidth=4
+*.pm		whitespace=space-before-tab,trailing-space,tabwidth=4
 *.po		whitespace=space-before-tab,trailing-space,tab-in-indent,-blank-at-eof
-*.sgml		whitespace=space-before-tab,trailing-space,tab-in-indent,-blank-at-eol
+*.sgml		whitespace=space-before-tab,trailing-space,tab-in-indent
 *.x[ms]l	whitespace=space-before-tab,trailing-space,tab-in-indent
 
 # Avoid confusing ASCII underlines with leftover merge conflict markers
@@ -17,18 +18,18 @@ src/backend/utils/Gen_dummy_probes.pl.prolog	whitespace=-blank-at-eof
 
 # Test output files that contain extra whitespace
 *.out					-whitespace
-contrib/*/output/*.source		-whitespace
-src/pl/plpgsql/src/output/*.source	-whitespace
-src/test/regress/output/*.source	-whitespace
 src/interfaces/ecpg/test/expected/*	-whitespace
-src/interfaces/libpq/test/expected.out	whitespace=-blank-at-eof
 
 # These files are maintained or generated elsewhere.  We take them as is.
 configure				-whitespace
 ppport.h				-whitespace
+src/backend/jit/llvm/SectionMemoryManager.cpp		-whitespace
+src/backend/jit/llvm/SectionMemoryManager.LICENSE	-whitespace
 src/backend/regex/COPYRIGHT		-whitespace
-src/backend/regex/re_syntax.n		-whitespace
 src/backend/snowball/libstemmer/*.c	-whitespace
 src/backend/utils/mb/Unicode/*-std.txt	-whitespace
+src/include/jit/SectionMemoryManager.h	-whitespace
 src/include/snowball/libstemmer/*	-whitespace
 src/timezone/data/*			-whitespace
+src/tools/pg_bsd_indent/*		-whitespace
+src/tools/pg_bsd_indent/tests/*		-whitespace

.github/workflows/apache-rat-audit.yml

@@ -344,4 +344,4 @@ jobs:
           else
             echo "FAILURE: One or more checks failed"
             exit 1
-          fi
+          fi

.github/workflows/coverity.yml

@@ -106,6 +106,7 @@ jobs:
             --with-libxml \
             --with-lz4 \
             --with-openssl \
+            --with-zstd \
             --with-pam \
             --with-perl \
             --with-pgport=5432 \

.github/workflows/sonarqube.yml

@@ -109,6 +109,7 @@ jobs:
             --with-libxml \
             --with-lz4 \
             --with-openssl \
+            --with-zstd \
             --with-pam \
             --with-perl \
             --with-pgport=5432 \

.gitignore

@@ -1,3 +1,7 @@
+# This file contains ignores for build artifacts from standard builds.
+# Auxiliary files from local workflows, your preferred editor, etc. should
+# be ignored locally using $GIT_DIR/info/exclude or ~/.gitexclude.
+
 # Global excludes across all subdirectories
 *.o
 *.obj
@@ -73,4 +77,5 @@ lib*.pc
 /compile_commands.json
 /tmp_install/
 /.cache/
-/install/
+/install/
+/portlock/

GNUmakefile.in

@@ -115,7 +115,7 @@ $(call recurse,distprep coverage,doc src config contrib gpcontrib)
 # it's not built by default
 $(call recurse,clean,doc contrib gpcontrib src config)
 clean:
-	rm -rf tmp_install/
+	rm -rf tmp_install/ portlock/
 # Garbage from autoconf:
 	@rm -rf autom4te.cache/
 # leap over gpAux/Makefile into subdirectories to avoid circular dependency.
@@ -132,7 +132,7 @@ distclean maintainer-clean:
 	$(MAKE) -C config $@
 	$(MAKE) -C gpMgmt $@
 	$(MAKE) -C src $@
-	rm -rf tmp_install/
+	rm -rf tmp_install/ portlock/
 # Garbage from autoconf:
 	@rm -rf autom4te.cache/
 	rm -f config.cache config.log config.status GNUmakefile
@@ -173,7 +173,7 @@ check:
 	  . $(prefix)/cloudberry-env.sh && . $(top_builddir)/gpAux/gpdemo/gpdemo-env.sh && $(MAKE) -C $(top_builddir) installcheck; \
 	fi
 
-$(call recurse,check-world,src/test src/pl src/interfaces/ecpg contrib src/bin gpcontrib,check)
+$(call recurse,check-world,src/test src/pl src/interfaces/ecpg contrib src/bin gpcontrib src/tools/pg_bsd_indent,check)
 $(call recurse,checkprep,  src/test src/pl src/interfaces/ecpg contrib src/bin gpcontrib)
 
 # This is a top-level target that runs "all" regression test suites against
@@ -202,6 +202,7 @@ endif
 ICW_TARGETS += gpcontrib gpMgmt/bin
 
 $(call recurse,installcheck-world,src/test src/pl src/interfaces/ecpg $(ICW_TARGETS) src/bin,installcheck)
+
 $(call recurse,install-tests,src/test/regress,install-tests)
 
 # GPDB: Postgres disables the SSL tests during ICW because of the TCP port that

Makefile

@@ -17,8 +17,14 @@ all:
 
 all check install installdirs installcheck installcheck-parallel uninstall clean distclean maintainer-clean dist distcheck world check-world install-world installcheck-world installcheck-resgroup installcheck-resgroup-v2:
 	@if [ ! -f GNUmakefile ] ; then \
+	   if [ -f INSTALL ] ; then \
+	     INSTRUCTIONS="INSTALL"; \
+	   else \
+	     INSTRUCTIONS="README.git"; \
+	   fi; \
 	   echo "You need to run the 'configure' program first. See the file"; \
-	   echo "'INSTALL' for installation instructions." ; \
+	   echo "'$$INSTRUCTIONS' for installation instructions, or visit: " ; \
+	   echo "<https://www.postgresql.org/docs/devel/installation.html>" ; \
 	   false ; \
 	 fi
 	@IFS=':' ; \

aclocal.m4

@@ -1,12 +1,10 @@
 dnl aclocal.m4
-m4_include([config/ac_func_accept_argtypes.m4])
 m4_include([config/ax_pthread.m4])
 m4_include([config/c-compiler.m4])
 m4_include([config/cxx-compiler.m4])
 m4_include([config/c-library.m4])
 m4_include([config/check_decls.m4])
 m4_include([config/general.m4])
-m4_include([config/libtool.m4])
 m4_include([config/llvm.m4])
 m4_include([config/perl.m4])
 m4_include([config/pkg.m4])