Skip to content

grpc.service_config timeout causing DEADLINE_EXCEEDED error too early #40633

@larashores

Description

@larashores

What version of gRPC and what language are you using?

Python grpc==1.74.0

What operating system (Linux, Windows,...) and version?

Mac Sequoia 15.6.1

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.13

What did you do?

I am using grpc.service_config to set a 2s timeout for a GRPC unary_unary call. The call on the server includes a 1s delay.

import asyncio
import contextlib
import json
import logging

import grpc

formatter = logging.Formatter("%(asctime)s: %(message)s")
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logging.root.addHandler(handler)
logging.root.setLevel(logging.INFO)

logger = logging.getLogger(__name__)


async def MyMethod(request: bytes, context: grpc.aio.ServicerContext) -> bytes:
    logger.info("Server Received: %s", repr(request))
    await asyncio.sleep(1)
    response = b"Goodbye, world!"
    logger.info("Server Sending: %s", repr(response))
    return response


@contextlib.asynccontextmanager
async def server():
    server = grpc.aio.server()
    server.add_insecure_port("[::]:9111")
    server.add_generic_rpc_handlers(
        (
            grpc.method_handlers_generic_handler(
                "MyService", {"MyMethod": grpc.unary_unary_rpc_method_handler(MyMethod)}
            ),
        )
    )
    logger.info("Server Starting")
    await server.start()

    logger.info("Server Started")
    try:
        yield
    finally:
        await server.stop(None)
        logger.info("Server Stopped")


async def client():
    channel = grpc.aio.insecure_channel(
        "localhost:9111",
        options=[
            (
                "grpc.service_config",
                json.dumps(
                    {
                        "methodConfig": [
                            {
                                "name": [{"service": "MyService"}],
                                "timeout": "2s",
                            }
                        ]
                    }
                ),
            )
        ],
    )
    callable = channel.unary_unary("/MyService/MyMethod")
    request = b"Hello World"

    logger.info("Client Sending: %s", repr(request))

    try:
        response = await callable(request)
    except grpc.aio.AioRpcError as exc:
        if exc.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
            logger.exception("Client Timeout")
            return
        raise
    
    logger.info("Client Received: %s", repr(response))


async def main():
    async with server():
        await client()


if __name__ == "__main__":
    asyncio.run(main())

What did you expect to see?

I expect that this code runs without error, as the 2s timeout on the client should be more than enough to accommodate the 1s timeout on the server.

What did you see instead?

Instead, the call always fails with a DEADLINE_EXCEEDED error. See the example logs below.

2025-09-05 14:48:44,172: Server Starting
2025-09-05 14:48:44,172: Server Started
2025-09-05 14:48:44,172: Client Sending: b'Hello World'
2025-09-05 14:48:44,173: Server Received: b'Hello World'
2025-09-05 14:48:44,228: Client Timeout
Traceback (most recent call last):
  File ".../server.py", line 72, in client
    response = await callable(request)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.13/site-packages/grpc/aio/_call.py", line 327, in __await__
    raise _create_rpc_error(
    ...<2 lines>...
    )
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.DEADLINE_EXCEEDED
        details = "Deadline Exceeded"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4}"
>
2025-09-05 14:48:44,229: Server Stopped

The DEADLINE_EXCEEDED error occured <100ms after the call was made, even though the timeout was set to 2s. Setting the timeout to >3s seems to be enough to prevent the timeout errors.

2025-09-05 14:52:56,322: Server Starting
2025-09-05 14:52:56,322: Server Started
2025-09-05 14:52:56,322: Client Sending: b'Hello World'
2025-09-05 14:52:56,323: Server Received: b'Hello World'
2025-09-05 14:52:57,324: Server Sending: b'Goodbye, world!'
2025-09-05 14:52:57,325: Client Received: b'Goodbye, world!'
2025-09-05 14:52:57,326: Server Stopped

Additionally, removing the timeout from the grpc.service_config and adding it to await callable(request, timeout=2) also fixes the issue. But I'd rather configure this globally in the config.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions