Revamp documentation

[asingamaneni]

Description

Overhaul the MkDocs documentation site with a clearer information architecture and expanded coverage:

Navigation & Structure:

• Reorganize mkdocs.yml nav into logical sections: Getting Started, User Guide, Examples, Developer Guide, and Python API
• Clean up API reference labels (remove underscores, fix capitalization)
• Remove navigation.expand for cleaner sidebar behavior

New Pages:

• Configuration Reference, Writers, Error Table, Secrets Backend, Rule Validation, FAQ & Troubleshooting, Example Notebooks
• Teams and Zoom notification guides

Content Improvements:

• Rewrite index.md with Mermaid sequence diagram, architecture table, and clearer feature overview
• Expand file_based_rules.md, quickstart.md, and why_spark_expectations.md
• Fix config key references in README (se_notifications_email_to_other_mail_id, se_enable_query_dq_detailed_result, se_enable_agg_dq_detailed_result)

Cleanup:

• Remove obsolete pages (github_copilot_chat_commands, key_prompts_for_repo, configurations_file, databricks_setup_guide)
• Relax mkdocs dependency pins in pyproject.toml to allow minor upgrades

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.56%. Comparing base (22d5ef8) to head (2ee38ab).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #302   +/-   ##
=======================================
  Coverage   98.56%   98.56%           
=======================================
  Files          32       32           
  Lines        3547     3547           
=======================================
  Hits         3496     3496           
  Misses         51       51           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.