Fixed #30200 -- Add support for using indexes in update() for ArrayFields. #19166

Jacob1507 · 2025-02-12T15:12:09Z

Trac ticket number

ticket-30200

Branch description

Add support for using indexes in update() for ArrayFields.

Checklist

This PR targets the main branch.
The commit message is written in past tense, mentions the ticket number, and ends with a period.
I have checked the "Has patch" ticket flag in the Trac system.
I have added or updated relevant tests.
I have added or updated relevant docs, including release notes if applicable.
I have attached screenshots in both light and dark modes for any UI changes.

Jacob1507 · 2025-02-12T15:23:51Z

Hi,

I was tweaking and testing some ideas and this is what I came up with. I would like to hear from you, if this is heading in right direction.

…elds.

jacobtylerwalls

Thanks for this PR! I have initial comments. I think we will want more input on whether ArrayField should implement as_sql as discussed below.

As you may already know, this will also need a release note in 6.1.txt.

You may be interested to see the related PR for JSONField (#18758). Because it is function-based, via JSON_SET, it only has to deal with right-hand sides, thus avoiding touching SQLUpdateCompiler and allowing assigning expressions targeting paths to model instances.

If we preferred, we could pursue a right-hand-side-only approach with concatenation via ||, e.g. before || new_value || after. I'd like to hear more views on this.

jacobtylerwalls · 2025-09-30T16:07:24Z

tests/postgres_tests/test_array.py

+    def test_update_on_db_value(self):
+        instance = NullableIntegerArrayModel(field=[1, 2], order=3)
+        instance.save()
+        NullableIntegerArrayModel.objects.update(field__0=F("order"))


It would be good to add a test where the update expression indexes an array like F("field__0") + 1 or such.

jacobtylerwalls · 2025-09-30T16:08:52Z

tests/postgres_tests/test_array.py

+        instance.refresh_from_db()
+        self.assertEqual(instance.field, [1, 2, 3, 4])
+
+    @unittest.expectedFailure


You may be waiting for feedback before polishing this, but when returning to this, you can remove these decorators and instead assert that the appropriate python error is raised. (At least one of these is failing at the database layer.)

jacobtylerwalls · 2025-09-30T16:18:24Z

tests/postgres_tests/test_array.py

+        instance.refresh_from_db()
+        self.assertEqual(instance.field, [0, 2, 3, 4])
+
+    def test_update_last(self):


We should also test slicing with negative indexes.

jacobtylerwalls · 2025-09-30T17:12:40Z

tests/postgres_tests/test_array.py

+        IntegerArrayModel.objects.update(
+            field__0_2=[-1, -2],
+            field__2_4=[-3, -4],
+            field__4_6=[-5, -6],


Nothing fails when I provide an array with more than two elements here.

jacobtylerwalls · 2025-09-30T17:20:43Z

django/contrib/postgres/fields/array.py

+        elif isinstance(array_slice, (list, tuple)):
+            return SliceTransformFactory(*array_slice)


I think we will want to observe self.size somewhere to make it harder to evade size by passing a slice that creates additional elements.

jacobtylerwalls · 2025-09-30T17:22:38Z

django/contrib/postgres/fields/array.py

        if hasattr(self.base_field, "from_db_value"):
            self.from_db_value = self._from_db_value
        super().__init__(**kwargs)
+        self.allow_lookups_on_update = True


I wonder if we can just rely on hasattr(field, "as_sql") instead of using this new flag.

jacobtylerwalls · 2025-09-30T17:48:37Z

django/contrib/postgres/fields/array.py

+        obj_copy.lookups = [self._prep_array_slice(lookup) for lookup in lookups]
+        return obj_copy
+
+    def as_sql(self, compiler, connection):


I was initially surprised to see a Field become a compilable.

Q1.) If it's going to be a compilable, I think it needs to implement get_source_expressions, see above link.

Q2.) Its compilability only makes sense for updates. Is there a way to reuse the compilables we already have, like IndexTransform? Then we could have a method here that's a sibling to get_placeholder like get_transforms_for_update or similar, and call that instead of as_sql.

/cc @charettes

Simon commented on the previous attempt with some implementation hints.

like get_transforms_for_update or similar, and call that instead of as_sql.

The linked review mentions an as_update_sql(), which sounds good.

jacobtylerwalls · 2025-09-30T19:14:37Z

tests/postgres_tests/test_array.py

+    def test_error_raised_on_not_supported_range_slice(self):
+        IntegerArrayModel.objects.update(field__1_1_1="2")
+
+    @unittest.expectedFailure


We should also test invalid indices like \n that will be swallowed by int().

E.g. this should raise a nice python error instead of failing at the database layer:

IntegerArrayModel.objects.update(**{"field__\n": None})

jacobtylerwalls · 2025-09-30T19:23:36Z

django/contrib/postgres/fields/array.py

+            if isinstance(array_slice, int):
+                lhs += "[%s]"
+                params.append(array_slice)
+
+            elif isinstance(array_slice, (list, tuple)):
+                if array_slice[1] is None:
+                    lhs += "[%s:]"
+                    params.append(array_slice[0])
+                else:
+                    lhs += "[%s:%s]"
+                    params.extend(array_slice)


Following Q2 above, my impression is that IndexTransform and SliceTransform already do this; can we delegate to them?

jacobtylerwalls · 2025-09-30T19:33:31Z

django/contrib/postgres/fields/array.py

+        if not isinstance(value, (list, tuple)):
+            return "%s"


I don't think we can do a naked type-check on value, because we might have a Value, e.g. let's add a test for

IntegerArrayModel.objects.update(field=Value([4, 5, 6]))

On main I get:

SET "field" = '{4,5,6}'::int2[]::bigint[];

On the PR I get:

SET "field" = '{4,5,6}'::int2[];

Perhaps one of the other test models provides a field where this would truly matter (and easier to assert against). We could add this test coverage in a separate commit.

Jacob1507 changed the title ~~Fixed #30200 - Add support for using indexes in update() for ArrayFields.~~ Fixed #30200 -- Add support for using indexes in update() for ArrayFields. Feb 12, 2025

Jacob1507 force-pushed the ticket_30200 branch 7 times, most recently from 3242899 to 9ed970e Compare February 16, 2025 13:12

Fixed #30200 -- Add support for using indexes in update() for ArrayFi…

11c132f

…elds.

Jacob1507 force-pushed the ticket_30200 branch from 9ed970e to 11c132f Compare February 17, 2025 18:26

jacobtylerwalls reviewed Sep 30, 2025

View reviewed changes

		elif isinstance(array_slice, (list, tuple)):
		return SliceTransformFactory(*array_slice)

Uh oh!

Fixed #30200 -- Add support for using indexes in update() for ArrayFields. #19166

Are you sure you want to change the base?

Fixed #30200 -- Add support for using indexes in update() for ArrayFields. #19166

Conversation

Jacob1507 commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Trac ticket number

Branch description

Checklist

Uh oh!

Jacob1507 commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacobtylerwalls left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jacobtylerwalls Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jacobtylerwalls Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Jacob1507 commented Feb 12, 2025 •

edited

Loading

Jacob1507 commented Feb 12, 2025 •

edited

Loading

jacobtylerwalls left a comment •

edited

Loading

jacobtylerwalls Sep 30, 2025 •

edited

Loading

jacobtylerwalls Sep 30, 2025 •

edited

Loading