[CORE-16880] kafka: autocreate internal topics with correct properties

[WillemKauf]

The auto create topics path would always create topics with default cluster configurations. This is very incorrect for certain special topics such as _schemas, __consumer_offsets, or our internal _redpanda.audit_log topic.

Pull these topic's special configs into publicly available accessors, and use these when autocreating these topics to ensure we are constructing these topics with the correct properties.

Fixes #31213

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v26.1.x
• v25.3.x
• v25.2.x

Release Notes

Bug Fixes

• Fixes a bug in which internal topics could be auto-created with the wrong topic properties

[CLAassistant]

All committers have signed the CLA.

[Copilot]

Pull request overview

This PR fixes Kafka metadata-driven auto-creation so that internal topics (e.g. _schemas, __consumer_offsets, _redpanda.audit_log) are created with subsystem-specific configurations rather than cluster-default topic properties, preventing silent misconfiguration when auto_create_topics_enabled=true.

Changes:

• Special-case internal topics in the Kafka metadata auto-create path to use the correct topic configurations (and delegate __consumer_offsets creation to group_initializer).
• Factor out canonical internal topic configurations/properties into reusable helpers (schema_registry_topic_configuration, audit_log_topic_properties, cluster::internal_topic_replication).
• Add coverage via both ducktape and C++ fixture tests to validate internal-topic auto-creation behavior.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/rptest/tests/internal_topic_protection_test.py	Adds ducktape coverage asserting internal topics auto-created via Metadata get correct partitions/RF/configs.
src/v/security/audit/client.cc	Uses shared audit_log_topic_properties() accessor when creating the audit topic.
src/v/security/audit/BUILD	Adds new audit topic helper sources/headers to the audit library target.
src/v/security/audit/audit_log_topic.h	Introduces public accessor for intended audit log topic properties.
src/v/security/audit/audit_log_topic.cc	Implements audit log topic properties (delete policy + 7-day retention).
src/v/pandaproxy/schema_registry/service.cc	Uses kafka::schema_registry_topic_configuration() helper instead of locally building properties.
src/v/pandaproxy/schema_registry/kafka_client_transport.cc	Prevents metadata probe from auto-creating _schemas by setting allow_auto_topic_creation=false.
src/v/kafka/server/tests/metadata_test.cc	Adds fixture test ensuring internal topics auto-created by metadata have correct configs.
src/v/kafka/server/handlers/topics/types.h	Exposes schema_registry_topic_configuration() helper in the kafka topics types API.
src/v/kafka/server/handlers/topics/types.cc	Implements schema registry internal topic configuration helper.
src/v/kafka/server/handlers/metadata.cc	Special-cases internal topics during metadata-driven auto-create to avoid cluster defaults.
src/v/kafka/server/group_initializer.cc	Centralizes internal-topic RF selection via cluster::internal_topic_replication().
src/v/cluster/members_table.h	Declares cluster::internal_topic_replication() helper.
src/v/cluster/members_table.cc	Implements RF helper based on internal_topic_replication_factor vs node count.
src/v/cluster/controller.cc	Reuses cluster::internal_topic_replication() in controller::internal_topic_replication().
src/v/cluster/BUILD	Adds config dependency to the members_table target for the new helper implementation.

[vbotbuildovich]

CI test results

test results on build#87585

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	NodeWiseRecoveryTest	test_recovery_local_data_missing	{"wait_for_final_manifest_uploads": false}	integration	https://buildkite.com/redpanda/redpanda/builds/87585#019f8b7a-4b23-43e7-b3b4-55906392cdb1	17/21	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0757, p0=0.1898, reject_threshold=0.0100. adj_baseline=0.2104, p1=0.3672, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeWiseRecoveryTest&test_method=test_recovery_local_data_missing
FLAKY(PASS)	IdempotentProducerRecoveryTest	test_java_client_recovery_on_producer_eviction	null	integration	https://buildkite.com/redpanda/redpanda/builds/87585#019f8b78-e89f-4a6a-a570-fe087aaaae9d	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0154, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=IdempotentProducerRecoveryTest&test_method=test_java_client_recovery_on_producer_eviction

[WillemKauf]

/backport v26.2.x

[vbotbuildovich]

/backport v26.1.x

[vbotbuildovich]

/backport v25.3.x

[vbotbuildovich]

/backport v25.2.x

[vbotbuildovich]

Failed to create a backport PR to v25.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-31234-v25.2.x-558 remotes/upstream/v25.2.x
git cherry-pick -x 5f2855f238 f2d1854bb6 932c5bc151 2be356fa08 f6ac88aa12

Workflow run logs.