proxy that will intercept TLS requests and add auth header

[ananth-sivam]

Hi,
I want to use proxy.py (version 2.4.3) for my application(custom plugin) which would receive API request of https, which will be validated and then request sent to the target endpoint.

"So basically our use case is a proxy that will intercept TLS requests and add auth header. The response from the target endpoint will be cached"

Please help by directing me to achieve this.

[noorul]

@abhinavsingh Any thoughts on this?

[abhinavsingh]

@ananth-sivam @noorul You can certainly write a custom plugin for this use case. Your plugin can:

1. Extends HttpProxyBasePlugin.handle_client_request method to add necessary auth headers.

2. For caching, you want to accumulate chunks of responses received from upstream. This can be done by extending HttpProxyBasePlugin.handle_upstream_chunk method.

   a) FWIW, there is already a CacheResponsePlugin which caches the responses on disk. I'll recommend to enable it once locally and study/re-use/extend over it's logic to accumulate response chunks for your caching needs.

   b) Note, CacheResponsePlugin doesn't serve out of cache, which is intentional. Serving out of cache is something you will have to implement within your plugin logic.

Hope this helps, let me know.

[noorul]

@abhinavsingh After intercepting the incoming connection shouldn't we create an upstream TLS connection?

[abhinavsingh]

It's not necessary, because proxy.py will do it for you. Plugins are essentially hooks into lifecycle of a client requests. If no plugin/hook override is provided, proxy.py will continue to work in a default fashion i.e. act like a simple proxy server (with or without TLS interception based upon configuration).

By adding plugin/hooks, we get a chance to override the behaviour of default lifecycle workflow. I'll recommend do check doc string for plugin base classes. Doc string will explain how to override/customise a particular event in the lifecycle e.g. handle_client_request, on_upstream_chunk, ..... etc

[noorul]

@abhinavsingh Is there any MITM sample for reference?

[abhinavsingh]

Yes you can extend over example plugins. Check readme for sample usage instructions here https://github.com/abhinavsingh/proxy.py#tls-interception

[ananth-sivam]

@abhinavsingh

#1. How do we overwrite request-timeout, connect-timeout on the request to upstream server?
i.e is it possible to provide values to the attributes(timeout/verify/...) in the request to upstream server
#2. Is there a way to use proxy lifecycle for retry mechanism
i.e. If on handle_upstream_chunk, we get a response that asks us to retry the request with updated body/auth to Upstream server

[abhinavsingh]

Apologies for missing out on this.

1. You can provide global values for these parameters, but likely you want a way to override per request. Is my understanding correct? If yes, unfortunately, there is no easy-peasy way to get this working, since config object is shared across all requests. However, we could easily add an abstract method which can optionally override global flag values (before request is handled by core).

2. No currently there is no such mechanisms for retries within handle_upstream_chunk, BUT, within your plugin code, once request has finished, you can inspect it to detect whether the request must be retried. If it must be retried, currently there is no convenient provision to retry a request within proxy.py framework. So your best bet is to just make the request within another asyncio loop and return the response as expected. Having said that, we could easily make ways for such retry scenarios.

If I have misunderstood your query, feel free to clarify the same to me.

[ananth-sivam]

Your understanding was Correct,
#1. want a way to override per request params. I still could override the flags.timeout param from before_upstream_connection method. But it doesn't differentiate the connect timeout vs read timeout. It just looks for the inactivity on both client and server side.
#2. mechanisms for retries within handle_upstream_chunk - Thanks, we already have a mechanism for the retry. We just wanted to be part of proxy lifecycle so that we do not have extra code

[ananth-sivam]

@abhinavsingh , I am able to override the flags.timeout on before_upstream_connection. But if timeout happens the client side is receiving
curl: (52) Empty reply from server
Or if python requests is used, then on timeout, ProxyError is thrown.
Is there a way in proxy.py, that we can capture this case and return 504 status code to the client?
Is there a way for us to respond a default response to client incase of any Exception on the proxy lib or the above case?

[abhinavsingh]

Please use --rewrite-host-header flag if you are using a http upstream. See latest commit #1492

[ananth-sivam]

@abhinavsingh, any suggestions

[ananth-sivam]

@abhinavsingh, any suggestions on #1333 (reply in thread)