[BACKPORT 2025.2][PLAT-18964] JInja2 version upgrade to 3.1.6

Summary:
Updated both python3_requirements and ynp_requirements
Original commit: 2dc8f01 / D48280

Test Plan: itest

Reviewers: anijhawan, nbhatia, nsingh, muthu

Reviewed By: muthu

Subscribers: nikhil, yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D48516

[BACKPORT 2025.2][#29240] DocDB: Don't parse empty string to CatalogE…

…ntityPB

Summary:
Catalog entry like Tables have required fields. For a Delete we do not populate the entry fields causing the call to `ParseFromString` to fail. If the input string is empty skip the parsing.

Original commit: af60489 / D48058

Test Plan: TEST_F(AdminCliTest, TestInsertDeleteTableEntry) {

Reviewers: #db-approvers, fizaa

Reviewed By: fizaa

Subscribers: mhaddad, ybase

Differential Revision: https://phorge.dev.yugabyte.com/D48493

[BACKPORT 2025.2][PLAT-19026] Fixing k8s restore with releases

Summary:
Node agent releases are being backed up as part of the filter in yb_platform_backup.sh and that causes issues when restoring to k8s without skip_old_files (like in isolated backup restore or continuous backup restore). We hit permission issues that releases already exist in destination. This diff fixes backup script to ignore the node-agent releases.
Original commit: 4551b16 / D48461

Test Plan: Take backup and verify node-agent releases are not present.

Reviewers: nsingh, vkumar, anijhawan, nbhatia

Reviewed By: nsingh

Subscribers: yugaware, nikhil

Differential Revision: https://phorge.dev.yugabyte.com/D48487

[BACKPORT 2025.2.0][#29056,#29057] YSQL: Fix intra-query memory leaks…

… in write queries

Summary:
This revision fixes various instances of intra-query memory leaks where memory is allocated on a per-tuple basis (but not in the per-tuple context) and is not cleared at the end of processing the tuple.
This result in transient memory spikes during long running write (INSERT/DELETE/UPDATE) queries. The allocated memory is currently freed en-masse at the end of the query, so the impact is short-lived.

A notable example is `YbComputeIndexExprOrPredicateAttrs` - a function that computes the columns composing expression and predicate indexes.
This function is invoked for every tuple that is updated by the query, resulting in ballooning memory allocations. This revision fixes the allocations in the execution layer.
While, this function is also invoked by the planner, the planner's allocations are one-time per query and do not need to be freed up immediately.
The results of this allocation can be cached in the future ([GH-29126](#29126)), to prevent recomputing the index columns for each tuple.

This revision also fixes an instance (in `ybplan.c`) where a log message produces the following warning when the number of indexes exceeds the number of columns:
```
WARNING:  01000: detected write past chunk end in PortalContext 0x114e74a40
LOCATION:  AllocSetFree, aset.c:1039
```

Original commit: ffbe86b / D47727

Test Plan:
Instrument memory allocations using the following queries:
```
-- In the backend (B1) running the long running write query
SET yb_tcmalloc_sample_period = '10kB';
SET yb_debug_log_docdb_requests TO false;
SELECT pg_backend_pid(); -- note this down as 'B1-pid'
<Run query>

-- In a different backend (B2), while the long running query is in progress, execute the following at regular intervals:
SELECT yb_log_backend_heap_snapshot(B1-pid);
SELECT * FROM pg_log_backend_memory_contexts(B1-pid);
```

Validate that:
 - Memory allocated to any of PG's memory contexts is not increasing with time
 - The heap snapshot mainly reports pggate allocations

The above exercise was run against the following long-running queries (each query affecting >= 500K rows)
 - INSERT into a table containing no indexes
 - INSERT into a table containing two indexes
 - INSERT ... ON CONFLICT ... DO NOTHING
 - INSERT ... ON CONFLICT ... DO UPDATE
 - UPDATE a table with no indexes modified by the update
 - UPDATE a table with two indexes modified
 - UPDATE a column of a table that references another table via a foreign key relationship
 - UPDATE involving expression/partial indexes
 - DELETE a table with no indexes
 - DELETE a table with two indexes

To validate that the log message doesn't produce the WARNING, run the following test with `log_min_messages=DEBUG2`:
```
./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressUpdateOptimized#schedule'
```

Reviewers: kfranz, jason, aagrawal

Reviewed By: jason

Subscribers: smishra, yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D48455

[BACKPORT 2025.2.0][PLAT-19026] Fixing k8s restore with releases

Summary:
Original commit: 4551b16 / D48461
Node agent releases are being backed up as part of the filter in yb_platform_backup.sh and that causes issues when restoring to k8s without skip_old_files (like in isolated backup restore or continuous backup restore). We hit permission issues that releases already exist in destination. This diff fixes backup script to ignore the node-agent releases.

Test Plan: Take backup and verify node-agent releases are not present.

Reviewers: nsingh, vkumar, anijhawan, nbhatia

Reviewed By: nsingh

Subscribers: nikhil, yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D48488

[BACKPORT-2.29.0.1081][CLOUDGA-30359] [instance capacity reservation]…

…[YBA:2.29.0.1081-b13]: Capacity reservation for Azure cluster is not getting created

Summary:
Original diff: 03ef146/D48480
Replaced provider name with cluster type in the naming pattern for capacity reservation groups/reservations

Test Plan: Create universe in azure

Reviewers: spothuraju, vkumar

Reviewed By: vkumar

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D48502

[#29070] YSQL: add SAOP merge to planning

Summary:
Introduce the planning part of a new feature "scalar array operation
merge".  The way SAOP merge works is to internally do a merge within an
index scan in order to return results sorted in an order that it
normally would not have been able to.

There are several factors that influence an index scan.  Effectively,
there are two possible sort outcomes: forward or backward.

- ScanDirection
  - BackwardScanDirection: results are sorted in backward order
  - NoMovementScanDirection: results are not sorted
  - ForwardScanDirection: results are sorted
- BitmapIndexScan: results are not sorted
- YB distinct index scan: depends on ScanDirection

SAOP merge introduces a new flavor of sort outcomes: forward/backward
on custom key columns (as long as sorting is in key column order and
unsorted columns up to the last key sort column are bound to either a
constant or list of constants through SAOP).  For example, given an
index (x int ASC, y int ASC, z int ASC), the following scans are
fulfillable:

- ORDER BY x, y, z
- ORDER BY x DESC, y DESC, z DESC
- WHERE x = 5 ORDER BY y, z
- WHERE x = 5 ORDER BY y DESC, z DESC

With SAOP merge, the following scans are now also fulfillable:

- WHERE x IN (5, 8) ORDER BY y, z
- WHERE x IN (5, 8) ORDER BY y DESC, z DESC

Under the hood, there will be two streams scanning the index, one for
x=5 and another for x=8, and they will be merge sorted on columns y and
z.  This part is not implemented in this commit as that is part of
execution (#29072); this commit is concerned with the planning part to
determine when SAOP merge is possible and form such a plan.
Proper costing of the plan is tracked by #29078.

To determine eligibility, index columns that are not bound equal to
constants and are not requested for sorting but have a usable SAOP are
considered good up until the cardinality of the cartesian product of
such SAOPs exceeds a new GUC yb_max_saop_merge_streams.  This GUC set to
zero disables SAOP from being considered, and that is the default for
now until the feature stabilizes.  Even when set greater than zero,
yb_enable_cbo must also be on in order for such index paths to be
considered by the CBO.

As part of this work, update EXPLAIN output to show new fields under
Index [Only] Scan in case SAOP merge is used:

- Merge Sort Key
- Merge Stream Key
- Merge Streams
- Merge Cond (only in verbose mode)

Note that "Merge Cond" duplicates entries in "Index Cond".  Perhaps it
can be changed in the future to deduplicate (remove entries from "Index
Cond").  Also, "Merge Streams" may not display an accurate number of
streams without resolving issue #29073.  All field names are subject to
rename before 2025.2.1 release.

Extra unused fields in YbSortInfo will be used for addressing #29072.

Test Plan:
On Almalinux 8:

    ./yb_build.sh fastdebug --gcc13 daemons initdb --skip-pg-parquet \
      --java-test TestPgRegressSaopMerge --tp 1 -n 100

Backport-to: 2025.2

Reviewers: amartsinchyk, patnaik.balivada, mtakahara

Reviewed By: amartsinchyk, patnaik.balivada

Subscribers: kyle.hailey, svc_phabricator, smishra, yql

Differential Revision: https://phorge.dev.yugabyte.com/D46214

[PLAT-18871] make SetupYNP reboot nodes before retries

Summary:
SetupYNP does not use the generally available node reboot based retry logic, so it's susceptible to failure when SSH is not responding.
This is resolved by making it a subclass of NodeTaskBase which has the onFailure hook.

Test Plan: During universe creation, temporarily stop/start a VM during node agent provisioning so that the task initially fails.

Reviewers: nsingh, svarshney

Reviewed By: nsingh

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D48513

[PLAT-18964] JInja2 version upgrade to 3.1.6

Summary: Updated both python3_requirements and ynp_requirements

Test Plan: itest

Reviewers: anijhawan, nbhatia, nsingh, muthu

Reviewed By: nsingh

Subscribers: yugaware, nikhil

Differential Revision: https://phorge.dev.yugabyte.com/D48280

[#29358,#28284] YSQL: Revert "Disallow disabling atomicity GUCs with …

…txn ddl"

Summary:
This reverts commit d893e01.

As part of d893e01 / D48297, we added safeguards to disallow disabling DDL atomicity infra GUCs when txn ddl is
enabled.

Presently, this leads to issues in scenarios when the user wants to disable both ddl atomicity and transactional ddl at the same time. Since we don't
have any guarantees on the ordering of GUC changes, the DDL atomicity GUC gets disabled first leading to the safeguard throwing an error since
transactional DDL is not yet disabled.

A few tests are failing as well due to this change. So we'll revert this and find another way to implement which requires a bit of discussion.

Test Plan: N/A

Reviewers: myang, aagrawal, sumukh.phalgaonkar

Reviewed By: sumukh.phalgaonkar

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D48496