Fixed #35629 -- Added support for async database connections and cursors. #18408

fcurella · 2024-07-24T20:04:49Z

Trac ticket number

Branch description

This would cover the new new_connection context managers, and provide an async cursor that the user could use.

The goal of this first phase is to give users a low-level async cursor that they can use for raw SQL queries

Manual transaction management, such as acommit and arollback is in scope, but transaction.atomic is out of scope and will be addressed in a future Pull Request.

Checklist

This PR targets the main branch.
The commit message is written in past tense, mentions the ticket number, and ends with a period.
I have checked the "Has patch" ticket flag in the Trac system.
I have added or updated relevant tests.
I have added or updated relevant docs, including release notes if applicable.
I have attached screenshots in both light and dark modes for any UI changes.

carltongibson · 2024-10-15T15:23:56Z

@fcurella Thanks for pushing this! I'm going to have a potter on it, and I'll see if @felixxm has a cycle too.

First glance it looks very minimal. 👍

bigfootjon

I'm not sufficiently familiar with the internals of the ORM/database layer to comment on much of the internal logic here, but the general shape of this code and approach seems to make sense to me

(left a few questions, nothing terribly blocking)

docs/topics/db/sql.txt

django/db/backends/utils.py

django/db/backends/base/base.py

django/db/backends/postgresql/base.py

LilyFirefly · 2024-10-22T10:00:18Z

django/db/backends/postgresql/base.py

Perhaps out of scope for this PR, but I don't think it's clear where this Cursor comes from. I think it's defined in psycopg3, but I'm not confident in that.

This should also be clarified in the ServerSideCursor docstring above too.

anything under Database. comes from psycopg2 or psycopg3.

In this specific case, ClientCursonMixin and AsyncClientCursor come from psycopg3

ServerSideCursor Is our custom class that derives from postgres' ClientCursorMixin and AsyncServerCursor/ServerCursor

LilyFirefly · 2024-10-22T10:10:13Z

django/db/backends/utils.py

Might this need to be async?

I've checked across all backends, and last_executed_query is a non-blocking method that basically only does string interpolation / decoding of a string that's already stored on the DBAPI cursor.

tests/db_utils/tests.py

django/db/__init__.py

Arfey · 2025-05-31T00:35:47Z

django/db/__init__.py

+
+        return self.conn
+
+    async def __aexit__(self, exc_type, exc_value, traceback):


I think I’ve found an issue with an async context manager — everything works fine until an asyncio.CancelledError is raised.

Here's a short example to reproduce it.

import asyncio class AsyncResource: async def __aenter__(self): print("Enter context") return self async def __aexit__(self, exc_type, exc, tb): print("Exiting context: start") await asyncio.sleep(2) print("Exiting context: end") async def task_that_uses_resource(): async with AsyncResource(): await asyncio.sleep(1) async def main(): task = asyncio.create_task(task_that_uses_resource()) # Let the task run a bit and then cancel it await asyncio.sleep(1.5) print("Cancelling the task...") task.cancel() try: await task except asyncio.CancelledError: print("Task was cancelled!") asyncio.run(main())

results

Enter context Exiting context: start Cancelling the task... Task was cancelled!

It’s clear why we don’t see Exiting context: end, but I’m not sure how to fix it quickly. This problem can lead to a leak of connections and incorrect nesting of connections due to pop_connection.

I'm trying to do something like that

# use sync context manager instead @contextmanager def independent_connection(self): active_connection = self.all(initialized_only=True) try: for conn in active_connection: del self[conn.alias] yield finally: # run async code in a separate coroutine without await asyncio.create_task(self.close_all()) for conn in active_connection: self[conn.alias] = conn

But it still doesn't cover every case. @fcurella, any thoughts on how we can to tackle this?

I wonder if shielding the exit logic would fix it? something like:

from asyncio import shield class new_connection: ... async def __aexit__(self, exc_type, exc_value, traceback): await shield(self._aexit(exc_type, exc_value, traceback)) async def _aexit(self, exc_type, exc_value, traceback): autocommit = await self.conn.aget_autocommit() if autocommit is False: if exc_type is None and self.force_rollback is False: await self.conn.acommit() else: await self.conn.arollback() await self.conn.aclose() await async_connections.pop_connection(self.using)

Could you write a test case to expose the bug?

I'm not entirely sure. Shielding seems to help with closing the connection, but I'm concerned it might not fix issues with broken nested ordering caused by pop_connection.

I'll try to write a unit test when I have some free time.

import asyncio from django.db import new_connection, async_connections, DEFAULT_DB_ALIAS from django.test import TransactionTestCase, skipUnlessDBFeature @skipUnlessDBFeature("supports_async") class NestedConnectionTest(TransactionTestCase): available_apps = ["async"] async def test_nesting_error1(self): async def aget_autocommit_new(): raise asyncio.CancelledError async with new_connection() as conn1: assert async_connections.count == 1 main_connection = async_connections.get_connection(DEFAULT_DB_ALIAS) aget_autocommit_old = conn1.aget_autocommit try: async with new_connection() as conn2: assert async_connections.count == 2 conn2.aget_autocommit = aget_autocommit_new except: pass conn1.aget_autocommit = aget_autocommit_old assert main_connection == async_connections.get_connection(DEFAULT_DB_ALIAS) assert async_connections.count == 1

This test contains monkey_patch, but it's clearly demonstrating the root of the problems. Is it helpful for you?

====================================================================== FAIL: test_nesting_error1 (async.test_async_connections.NestedConnectionTest.test_nesting_error1) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/m.havelya/work/django/env/lib/python3.13/site-packages/asgiref/sync.py", line 325, in __call__ return call_result.result() ~~~~~~~~~~~~~~~~~~^^ File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py", line 449, in result return self.__get_result() ~~~~~~~~~~~~~~~~~^^ File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/Users/m.havelya/work/django/env/lib/python3.13/site-packages/asgiref/sync.py", line 365, in main_wrap result = await awaitable ^^^^^^^^^^^^^^^ File "/Users/m.havelya/work/django/tests/async/test_async_connections.py", line 31, in test_nesting_error1 assert main_connection == async_connections.get_connection(DEFAULT_DB_ALIAS) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError

Thank you, that's really helpful!

We could wrap the __exit__ in a try ... finally block:

async def __aexit__(self, exc_type, exc_value, traceback): try: autocommit = await self.conn.aget_autocommit() if autocommit is False: if exc_type is None and self.force_rollback is False: await self.conn.acommit() else: await self.conn.arollback() finally: await self.conn.aclose() await async_connections.pop_connection(self.using)

We can get this problem during aclose call as well

robd003 · 2025-05-31T21:33:11Z

When this is finalized and merged to master will it also be added to the subsequent Django 5.2.x release?

dima-kov · 2025-08-28T08:51:47Z

Dear django team, what's your decision on this? Kindly reminder, a lot of people waiting for low level async db connection

fcurella · 2025-08-28T13:01:13Z

@dima-kov I’ve been swamped at work and couldn’t dedicate much time to it. I would definitely appreciate some help!

carltongibson · 2025-08-28T14:36:50Z

@dima-kov I think the way forward here is to pull this work into a third-party backend so we can all get trying it out.

This is on my list but I, too, haven't had spare capacity, yet.

Co-authored-by: Mykhailo Havelia <[email protected]>

Arfey · 2025-10-01T23:03:59Z

@dima-kov I think the way forward here is to pull this work into a third-party backend so we can all get trying it out.

This is on my list but I, too, haven't had spare capacity, yet.

Right now, it’s just a cursor implementation. To test it properly in a real project, we’ll need the ORM parts (transactions, get, update, delete, etc.), otherwise, it doesn’t make much sense. 😌

We already have contributors willing to finish it up, but it will need input from the core team for validation and review.

@carltongibson would it make sense to form a small group with 3-4 maintainers familiar with the ORM, together with contributors (django forum, discord, etc)? That way, if one person is busy, others could still provide guidance and keep things moving.

What do you think?

carltongibson · 2025-10-04T10:30:25Z

@Arfey yep, I think coordinating effort is good. (I've wanted to pick this up, but haven't had capacity. I'm totally happy for others to do so.)

My concern here is that it needs to be properly tested in production prior to merging. That means we need to give people an easy way to add it to their existing projects. Which means we need to extract it to a package. (I don't see another way there. Maybe there is one.)

There are guaranteed to be weird deadlocking issues (and similar) that folks run into when they start using this, and we need to fix issues and document the usage patterns, so that it's received well.

Arfey · 2025-10-05T17:43:35Z

@Arfey yep, I think coordinating effort is good. (I've wanted to pick this up, but haven't had capacity. I'm totally happy for others to do so.)

@carltongibson Got it. I'll put together a brief overview of this ticket for the Discord chat. Hopefully, we can form a working group 😌

My concern here is that it needs to be properly tested in production prior to merging. That means we need to give people an easy way to add it to their existing projects. Which means we need to extract it to a package. (I don't see another way there. Maybe there is one.)Got it. I'll put together a brief overview of this ticket for the Discord chat. Hopefully, we can form a working group 😌

I've been thinking about it. It's a really tricky part. Even an async connection required significant changes within Django. Extracting it into a separate extension would involve a lot of copy-pasting, and after testing it in production, merging it back into Django could be risky and might introduce bugs.

I see two options:

Use a separate database backend (which I partially introduced in Asynchronous implementation of a database connection #19576)
Use a feature flag, like we do with DJANGO_ALLOW_ASYNC_UNSAFE

carltongibson · 2025-10-05T18:47:57Z

Extracting it into a separate extension would involve a lot of copy-pasting

I don't really see this as a problem. Yes, there'll be some duplication. (But there's lots here that wouldn't be needed in a proof of concept IIRC. E.g. can we subclass the Postgres backend just to add the new api? Etc. It doesn't seem intractable.)

If we put it in an external package we can ask people to pick it up as soon as it's available. (Rather than waiting until Sept 26 for Django 6.1.)

We can then test and fix issues on a much quicker cycle than waiting for changes in Django.

I'm pretty sure we can correctly merge again once the edge cases are worked out. (We'll then be doing so knowing we have a battle-tested approach.)

(Just personally, I could easily install and define an alias to try this on the work project, but to wait for Django 6.1 is going to be an age.)

Arfey · 2025-10-06T10:50:22Z

@carltongibson, probably I'm overthinking too much, excuse me 😌 Maybe you’re right.

There are a lot of places that we can’t reuse from Django without rewriting the entire code base. A quick example is get_new_connection or transaction.atomic. They both have a lot of entire logic which mixed with sync calls. As a result, we can’t reuse it coz we’ve gotta replace sync call to async. In this case, the only way is full coping and pasting all methods like this.

async def aget_new_connection(self, conn_params):
	  # copy code ...

	  if self.apool:
	      await self.apool.open()
	      connection = await self.apool.getconn()
	  else:
	      connection = await self.Database.AsyncConnection.connect(**conn_params)

	  # copy code ...

	  return connection

Very often, we can’t reuse code by extending a class. So we will have a lot of duplication. We need to avoid any changes inside duplication code and sync Django’s changes with our extension. I think it’s tricky 🙂

Anyway, your proposal has a lot of benefits (especially when we are talking about speed). Let’s try. I’m gonna prepare POC.

Is it possible to add to Django yield statements to the DB backend class? ☺️ It significantly decreases the amount of duplication.

from this

def _close(self):
		if self.connection is not None:
		    with self.wrap_database_errors:
		       self.connection.close()

to this

@sync_async_method_adapter    # <---- sync_async_method_adapter
def _close(self):
		if self.connection is not None:
		    with self.wrap_database_errors:
		        yield self.connection.close()      # <---- yield

django/django/utils/sync_async.py

Lines 25 to 76 in 5235448

    
           def run_sync_generator(gen): 
        
               """ 
        
               Run a generator-based coroutine in a synchronous context. 
        
               """ 
        
               res = None 
        
               try: 
        
                   while True: 
        
                       try: 
        
                           res = gen.send(res) 
        
                       except Exception as e: 
        
                           res = gen.throw(e) 
        
               except StopIteration: 
        
                   pass 
        
               return res 
        
           def sync_async_method_adapter(fn): 
        
               """ 
        
               A decorator for methods that return a generator-based coroutine 
        
               (sync or async). It uses the class's `sync_async_adapter` 
        
               method to execute the coroutine depending on the context 
        
               (e.g., run_async_generator or run_sync_generator). 
        
               Example: 
        
                   class Base: 
        
                       @sync_async_method_adapter 
        
                       def my_method(self): 
        
                           yield self.client().get() 
        
                   class Async(Base): 
        
                       sync_async_adapter = run_async_generator 
        
                       client = AsyncClient 
        
                   class Sync(Base): 
        
                       sync_async_adapter = run_sync_generator 
        
                       client = SyncClient 
        
               """ 
        
               @functools.wraps(fn) 
        
               def inner(self, *args, **kwargs): 
        
                   adapter = getattr(self.__class__, 'sync_async_adapter', None) 
        
                   if not callable(adapter): 
        
                       raise TypeError( 
        
                           f"{self.__class__.__name__} must " 
        
                           "define a 'sync_async_adapter' method" 
        
                       ) 
        
                   return adapter(fn(self, *args, **kwargs)) 
        
               return inner

carltongibson · 2025-10-06T11:00:41Z

Let’s try. I’m gonna prepare POC.

Super! 🏅

Just to be totally clear, I really want us to progress this feature. It's the last main bit of the async story. They'll be more to (refinements) follow but if we had this in place, we'd have a whole DB-to-wire story folks could use.

A quick example is get_new_connection or transaction.atomic. They both have a lot of entire logic which mixed with sync calls. As a result, we can’t reuse it coz we’ve gotta replace sync call to async. In this case, the only way is full coping and pasting all methods like this.

Gotcha.

It's still the case that pulling them out (as duplicates) lets us get going, and we can say "Can we merge this small part here?", which is easier to review and check for correctness.

I know this all feels like more work. But I've seen subtle problems with async come up again and again over these last number of years — the discussion about cancellation above is exactly of that kind — and I can't see a way for us to be really confident that we're have a solid (Django Quality™) approach without getting people to put it under stress in production and then report back. I hope that makes sense.

Arfey · 2025-10-06T12:09:25Z

Just to be totally clear, I really want us to progress this feature. It's the last main bit of the async story. They'll be more to (refinements) follow but if we had this in place, we'd have a whole DB-to-wire story folks could use.

Agree.

Just to confirm. Am I right that changing yield isn’t an option, and I should prepare a POC with full duplication for the cases where inheritance doesn’t work? 😌

carltongibson · 2025-10-06T12:26:09Z

Just to confirm. Am I right that changing yield isn’t an option, and I should prepare a POC with full duplication for the cases where inheritance doesn’t work? 😌

🤷 :)

I haven't had a chance to sit down with #19576 and really think it through. Maybe the sync_async_method_adapter makes sense (and duplication is a problem we've hit elsewhere…) but I would always favour the simpler change, followed by the refactoring. (Thus allowing them to be assessed separately.)

In general, I think "Here's the simplest thing we can do" is going to be easier to get moving.

Arfey · 2025-10-06T14:26:25Z

Understood. I'll create a simple MR adding yield to Django and build an extension on top of it. If the MR gets rejected, I'll copy the code into the extension instead.

dima-kov · 2025-10-06T14:35:00Z

If we put it in an external package we can ask people to pick it up as soon as it's available. (Rather than waiting until Sept 26 for Django 6.1.)

I might be missing something, but doesn't creating an external package feel like unnecessary overhead? Why not just publish it as a separate Git tag - e.g. 6.1a0 (or aorm.0?)- on GitHub so people can try it out easily?
Subsequent patches could just be new tags like 6.1a1, 6.1a2, and so on. This could even live on the fcurella/django, not to polute django/django

carltongibson · 2025-10-06T17:58:15Z

@Arfey even if such a patch is accepted immediately (which is unlikely) it wouldn't be available until Django 6.1 next September. The whole point here of our discussion is to get people being able to use async cursors now.

I was talking about this with the other Steering Council members today. We agreed this needs to be tested in production before we can merge it. (The example of being shown to handle high numbers of connections in production without causing issues was part of the discussion.) We can't get there without an external implementation first, as a pathway to getting this resolved.

Arfey · 2025-10-06T21:07:14Z

@carltongibson got it. I'll prepare a fully separated version.

The example of being shown to handle high numbers of connections in production without causing issues was part of the discussion.

Could u share some thoughts about testing it in production? Let's say we have a ready external implementation of an asynchronous cursor. What are my next steps? Do I need to install it and rewrite some parts of my project using raw SQL and execute it with an asynchronous cursor?

fcurella force-pushed the async-orm-cursor branch from ed91e56 to 83099c9 Compare July 24, 2024 21:34

fcurella changed the title ~~Fixed #35629. Async DB Connection and Cursor~~ Draft: Fixed #35629. Async DB Connection and Cursor Jul 24, 2024

fcurella force-pushed the async-orm-cursor branch 6 times, most recently from 3a3fc57 to e6bbcd1 Compare July 25, 2024 20:03

fcurella changed the title ~~Draft: Fixed #35629. Async DB Connection and Cursor~~ Fixed #35629. Async DB Connection and Cursor Jul 25, 2024

jaw9c mentioned this pull request Aug 1, 2024

Support for Async Django graphql-python/graphene-django#1394

Open

2 tasks

fcurella force-pushed the async-orm-cursor branch 5 times, most recently from 1b81214 to b42cc45 Compare August 9, 2024 13:47

fcurella force-pushed the async-orm-cursor branch from b42cc45 to 86158fb Compare August 19, 2024 16:22

fcurella force-pushed the async-orm-cursor branch from 86158fb to 3be837e Compare September 4, 2024 14:03

fcurella force-pushed the async-orm-cursor branch from 3be837e to 288b3c9 Compare September 18, 2024 13:33

fcurella force-pushed the async-orm-cursor branch from 288b3c9 to df58a82 Compare October 15, 2024 14:21

carltongibson requested review from carltongibson and felixxm October 15, 2024 15:22

bigfootjon reviewed Oct 22, 2024

View reviewed changes

docs/topics/db/sql.txt Outdated Show resolved Hide resolved

django/db/backends/utils.py Outdated Show resolved Hide resolved

django/db/backends/base/base.py Outdated Show resolved Hide resolved

LilyFirefly reviewed Oct 22, 2024

View reviewed changes

fcurella force-pushed the async-orm-cursor branch 2 times, most recently from 259d10c to 53fae35 Compare October 22, 2024 15:26

pauloxnet reviewed Oct 23, 2024

View reviewed changes

django/db/__init__.py Outdated Show resolved Hide resolved

fcurella force-pushed the async-orm-cursor branch 3 times, most recently from efafc58 to 49141bb Compare October 24, 2024 14:24

fcurella force-pushed the async-orm-cursor branch from 126f15a to fd3053b Compare May 23, 2025 14:10

Arfey reviewed May 31, 2025

View reviewed changes

carltongibson mentioned this pull request Jun 19, 2025

Asynchronous implementation of a database connection #19576

Open

6 tasks

fcurella and others added 8 commits September 26, 2025 09:47

Fixed #35629. Async DB Connection and Cursor

96a3fd4

Move async_connections._from_testcase to TestCase

5a70cc3

Reduce code duplication with connect_manager

845ae58

Implement ais_usable

2e34b5f

made pg_version and apg_version idemponent

5d90095

Update django/db/backends/utils.py

08ba324

Co-authored-by: Mykhailo Havelia <[email protected]>

Add missing for_async argument

9afa1ec

move supports_async to DatabaseFeatures class.

2f3dbce

fcurella force-pushed the async-orm-cursor branch from fd3053b to 2f3dbce Compare September 26, 2025 14:47

lint code

da20298

fcurella force-pushed the async-orm-cursor branch from 20104d6 to da20298 Compare September 26, 2025 14:58


		return self.conn

		async def __aexit__(self, exc_type, exc_value, traceback):

Uh oh!

Fixed #35629 -- Added support for async database connections and cursors. #18408

Are you sure you want to change the base?

Fixed #35629 -- Added support for async database connections and cursors. #18408

Uh oh!

Conversation

fcurella commented Jul 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Trac ticket number

Branch description

Checklist

Uh oh!

carltongibson commented Oct 15, 2024

Uh oh!

bigfootjon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LilyFirefly Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

fcurella Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LilyFirefly Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

fcurella Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Arfey May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fcurella Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

Arfey Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

Arfey Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fcurella Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

Arfey Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robd003 commented May 31, 2025

Uh oh!

dima-kov commented Aug 28, 2025

Uh oh!

fcurella commented Aug 28, 2025

Uh oh!

carltongibson commented Aug 28, 2025

Uh oh!

Arfey commented Oct 1, 2025

Uh oh!

carltongibson commented Oct 4, 2025

Uh oh!

Arfey commented Oct 5, 2025

Uh oh!

carltongibson commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

fcurella commented Jul 24, 2024 •

edited

Loading

fcurella Oct 23, 2024 •

edited

Loading

fcurella Oct 23, 2024 •

edited

Loading

Arfey May 31, 2025 •

edited

Loading

Arfey Jun 5, 2025 •

edited

Loading

Arfey Jun 6, 2025 •

edited

Loading

carltongibson commented Oct 5, 2025 •

edited

Loading

carltongibson commented Oct 6, 2025 •

edited

Loading

dima-kov commented Oct 6, 2025 •

edited

Loading