Python: updated python design and dev setup (#171)

* updated python design and dev setup

* updated dev setup

* updated dev setup

Author: eavanvalkenburg

docs/design/python-package-setup.md

@@ -27,10 +27,10 @@
 ### Sample getting started code
 ```python
 from typing import Annotated
-from agent_framework import Agent, ai_tool
+from agent_framework import Agent, ai_function
 from agent_framework.openai import OpenAIChatClient
 
-@ai_tool(description="Get the current weather in a given location")
+@ai_function(description="Get the current weather in a given location")
 async def get_weather(location: Annotated[str, "The location as a city name"]) -> str:
     """Get the current weather in a given location."""
     # Implementation of the tool to get weather
@@ -50,12 +50,12 @@ print(response)
 Overall the following structure is proposed:
 
 * agent-framework
-    * tier 0; components, will be exposed directly from `agent_framework`:
+    * core components, will be exposed directly from `agent_framework`:
         * (single) agents (includes threads)
         * tools (includes MCP and OpenAPI)
         * models/types (name tbd, will include the equivalent of MEAI for dotnet; content types and client abstractions)
         * logging
-    * tier 1; components, will be exposed from `agent_framework.<component>`:
+    * advanced components, will be exposed from `agent_framework.<component>`:
         * context_providers (tbd)
         * guardrails / filters
         * vector_data (vector stores and other MEVD pieces)
@@ -65,12 +65,12 @@ Overall the following structure is proposed:
         * utils (optional)
         * telemetry (could also be observability or monitoring)
         * workflows (includes multi-agent orchestration)
-    * tier 2; extensions 
-        * Extensions are any additional functionality that is useful for a user, to reduce friction they will imported in a similar way as tier 1, however the code for them will be in a separate package, so that they can be installed separately, they must have everything in a folder with the same name as the package, and without a `__init__.py` file in the root, so that they can be used as a namespace package.
+    * connectors; subpackages
+        * Subpackages are any additional functionality that is useful for a user, to reduce friction they will imported in a similar way as advanced components, however the code for them will be in a separate package, so that they can be installed separately, they must expose all public items, in their main `__init__.py` file, so that they can be imported from the main package without additional import levels.
+        In the main package a corresponding folder will be created, with a `__init__.py` file that lazy imports the public items from the subpackage, so that they can be exposed from the main package.
         * Some examples are:
-            * openai
-            * azure
-            * will be exposed through i.e. `agent_framework.openai` and `agent_framework.azure`
+            * azure (non LLM integrations)
+            * will be exposed through i.e. `agent_framework.azure`
             * anything other then a connector that we want to expose as a separate package, for instance:
                 * mem0 (memory management)
                 * would be exposed through i.e. `agent_framework.mem0`
@@ -79,7 +79,6 @@ Overall the following structure is proposed:
 * tests
 * samples
 * extensions
-    * openai
     * azure
     * ...
 
@@ -90,53 +89,48 @@ Internal imports will be done using relative imports, so that the package can be
 The resulting file structure will be as follows:
 
 ```plaintext
-agent_framework/
-    __init__.py
-    __init__.pyi
-    _agents.py
-    _tools.py
-    _models.py
-    _logging.py
-    context_providers.py
-    guardrails.py
-    exceptions.py
-    evaluations.py
-    utils.py
-    telemetry.py
-    templates.py
-    text_search.py
-    vector_data.py
-    workflows.py
-    py.typed
-extensions/
-    mem0/
-        agent_framework/
-            mem0/
-                __init__.py
-                _mem0.py
-                ...
-    redis/
-        ...
-    openai/
+packages/
+    main/
         agent_framework/
             openai/
                 __init__.py
-                _chat.py
-                _embeddings.py
-                ...
+                _chat_client.py
+                _shared.py
+                exceptions.py
+            __init__.py
+            __init__.pyi
+            _agents.py
+            _tools.py
+            _models.py
+            _logging.py
+            context_providers.py
+            guardrails.py
+            exceptions.py
+            evaluations.py
+            utils.py
+            telemetry.py
+            templates.py
+            text_search.py
+            vector_data.py
+            workflows.py
+            py.typed
         tests/
             unit/
-                test_openai_client.py
-                test_openai_tools.py
-                ...
+                test_types.py
             integration/
-                test_openai_integration.py
-        samples/ (optional)
-            ...
+                test_chat_clients.py
         pyproject.toml
         README.md
         ...
-    google/        
+    mem0/
+        agent_framework/
+            mem0/
+                __init__.py
+                _mem0.py
+                ...
+    redis/
+        ...
+    google/
         agent_framework/
             google/
                 __init__.py
@@ -156,13 +150,6 @@ extensions/
         README.md
         ...
     ...
-tests/
-    __init__.py
-    unit/
-        conftest.py
-        test_agents.py
-        test_types.py
-        ...
 samples/
     ...
 pyproject.toml
@@ -174,10 +161,10 @@ uv.lock
 
 We might add a template subpackage as well, to make it easy to setup, this could be based on the first one that is added.
 
-In the `DEV_SETUP.md` we will add instructions for how to deal with the path depth issues, especially on Windows, where the maximum path length can be a problem.
+In the [`DEV_SETUP.md`](../../python/DEV_SETUP.md) we will add instructions for how to deal with the path depth issues, especially on Windows, where the maximum path length can be a problem.
 
 #### Evolving the package structure
-For each of the tier 1 components, we have two reason why we may split them into a folder, with an `__init__.py` and optionally a `_files.py`:
+For each of the advanced components, we have two reason why we may split them into a folder, with an `__init__.py` and optionally a `_files.py`:
 1. If the file becomes too large, we can split it into multiple `_files`, while still keeping the public interface in the `__init__.py` file, this is a non-breaking change
 2. If we want to partially or fully move that code into a separate package.
 In this case we do need to lazy load anything that was moved from the main package to the subpackage, so that existing code still works, and if the subpackage is not installed we can raise a meaningful error.
@@ -197,14 +184,7 @@ agent_framework/
 
 ## Coding standards
 
-* We use google docstyles for docstrings. 
-* We use the following setup for ruff:
-```toml
-[tool.ruff]
-line-length = 120
-target-version = "py310"
-include = ["*.py", "*.pyi", "**/pyproject.toml", "*.ipynb"]
-preview = true
+Coding standards will be maintained in the [`DEV_SETUP.md`](../../python/DEV_SETUP.md) file.
 
 [tool.ruff.lint]
 fixable = ["ALL"]
@@ -281,16 +261,16 @@ The logging will be simplified, there will be one logger in the base package:
 * name: `agent_framework` - used for all logging in the abstractions and base components
 
 Each of the other subpackages for connectors will have a similar single logger.
-* name: `agent_framework.connectors.openai` 
-* name: `agent_framework.connectors.azure` 
+* name: `agent_framework.openai`
+* name: `agent_framework.azure`
 
 This means that when a logger is needed, it should be created like this:
 ```python
 from agent_framework import get_logger
 
 logger = get_logger()
 #or in a subpackage:
-logger = get_logger('agent_framework.connectors.openai')
+logger = get_logger('agent_framework.openai')
 ```
 The implementation should be something like this:
 ```python
@@ -300,18 +280,18 @@ import logging
 def get_logger(name: str = "agent_framework") -> logging.Logger:
     """
     Get a logger with the specified name, defaulting to 'agent_framework'.
-    
+
     Args:
         name (str): The name of the logger. Defaults to 'agent_framework'.
-    
+
     Returns:
         logging.Logger: The configured logger instance.
     """
     logger = logging.getLogger(name)
     # create the specifics for the logger, such as setting the level, handlers, etc.
     return logger
 ```
-This will ensure that the logger is created with the correct name and configuration, and it will be consistent across the package. 
+This will ensure that the logger is created with the correct name and configuration, and it will be consistent across the package.
 
 Further there should be a easy way to configure the log levels, either through a environment variable or with a similar function as the get_logger.
 
@@ -320,7 +300,7 @@ This will not be allowed:
 import logging
 
 logger = logging.getLogger(__name__)
-``` 
+```
 
 This is allowed but discouraged, if the get_logger function has been called at least once then this will return the same logger as the get_logger function, however that might not have happened and then the logging experience (in terms of formats and handlers, etc) is not consistent across the package:
 ```python
@@ -332,40 +312,9 @@ logger = logging.getLogger("agent_framework")
 #### Telemetry
 Telemetry will be based on OpenTelemetry (OTel), and will be implemented in the `agent_framework.telemetry` package.
 
-We should consider auto-instrumentation and provide an implementation of it to the OTel community.
-
-### Function definitions
-To make the code easier to use, we will be very deliberate about the ordering and marking of function parameters.
-This means that we will use the following conventions:
-* Only parameters that are fully expected to be passed and only if there are a very limited number of them, let's say 3 or less, can they be supplied as positional parameters (still with a keyword, _almost_ never positional only).
-* All other parameters should be supplied as keyword parameters, this is especially important to configure correctly when using Pydantic or dataclasses.
-* If there are multiple required parameters, and they do not have a order that is common sense, then they will all use keyword parameters.
-* If we use `kwargs` we will document how and what we use them for, this might be a reference to a outside package's documentation or an explanation of what the `kwargs` are used for.
-* If we want to combine `kwargs` for multiple things, such as partly for a external client constructor, and partly for our own use, we will try to keep those separate, by adding a parameter, such as `client_kwargs` with type `dict[str, Any]`, and then use that to pass the kwargs to the client constructor (by using `Client(**client_kwargs)`), while using the `**kwargs` parameters for other uses, which are then also well documented.
-
-### Attributes vs inheritance
+We will also add headers with user-agent strings where applicable, these will include `agent-framework-python` and the version.
 
-When the parameters are the same except for one, we will use attributes, instead of inheritance, to minimize the conceptual overhead of understanding the code. Off course there are exceptions and these things will be decided on a case by case basis, but the general rule is that if the parameters are the same, we will use attributes.
-```python
-# ✅ preferred
-from agent_framework import ChatMessage
-user_msg = ChatMessage(
-    role="user",
-    content="Hello, world!"
-)
-asst_msg = ChatMessage(
-    role="assistant",
-    content="Hello, world!"
-)
-# ❌ not preferred
-from agent_framework import UserMessage, AssistantMessage
-user_msg = UserMessage(
-    content="Hello, world!"
-)
-asst_msg = AssistantMessage(
-    content="Hello, world!"
-)
-```
+We should consider auto-instrumentation and provide an implementation of it to the OTel community.
 
 ### Build and release
 The build step will be done in GHA, adding the package to the release and then we call into Azure DevOps to use the ESRP pipeline to publish to pypi. This is how SK already works, we will just have to adapt it to the new package structure.