Feature/sckan-394-395-396 - Population set - Backend, UI, and Admin UI #420

D-GopalKrishna · 2025-02-05T13:16:14Z

Task description:

https://metacell.atlassian.net/browse/SCKAN-394
https://metacell.atlassian.net/browse/SCKAN-395
https://metacell.atlassian.net/browse/SCKAN-396 - (not the third part) - that is taken care in a separate PR.

More on the PR:

The Backend changes:

Add populationset - models, URLs, views, and serializers
add another field to KS - has_statement_been_exported
along with the migrations; a custom migration - migrate_has_statement_been_exported - to fill the field for everything that is exported currently
Two constraints:
- Do not allow to go to EXPORTED if doesn't have populationset - by adding condition

conditions=[
            ConnectivityStatementStateService.is_valid,
            ConnectivityStatementStateService.has_populationset,
        ],

- Do not allow to change the populationset for a KS - if it had been ever in exported state

    def validate_population_change(self):
        if self.pk and self.has_statement_been_exported:
            original = ConnectivityStatement.objects.get(pk=self.pk)
            if original.population != self.population:
                raise ValidationError(
                    "Cannot change population set after the statement has been exported."
                )

The Composer UI changes:

update the genapi.sh
Add the copiedUISchema.population_id for - populationset
- make it read only if isDisabled: statement.has_statement_been_exported is true
make a call to populations and set it using PopulationService - and call it onLogin.

The Admin UI changes:

Add PopulationSet to the admin UI.

…odels

…xported read only

…pdating new field - has_statement_been_exported

… doesn't have EXPORT available transition

…opulationSet

…opulationService

backend/composer/models.py

…ment_been_exported migration

backend/composer/services/cs_ingestion/helpers/statement_helper.py

backend/composer/services/export/helpers/export_batch.py

backend/composer/utils.py

backend/composer/models.py

backend/composer/services/state_services.py

backend/composer/migrations/0075_migrate_has_statement_been_exported.py

backend/composer/models.py

backend/composer/services/cs_ingestion/helpers/statement_helper.py

backend/composer/services/export/helpers/export_batch.py

afonsobspinto · 2025-02-06T00:07:31Z

frontend/src/index.tsx

-                  <App />
-                </Provider>
-              );
+          populations.setPopulations().then(() => {


Not related with this PR, but I don't love this nested callbacks. I think most of these fetches are independent, if that's the case we can use Promise.all instead to have them happening in parallel. It can be a tecnical debt card for later, cc @ddelpiano

…t state change

…tatement helper

…izer and refactor population change validation in ConnectivityStatement forms

…lation

…exported directly on instance

…sitioning to exported state

…atement constraints

…d utility to extract population set from statement ID

…label

backend/composer/models.py

afonsobspinto · 2025-02-14T01:31:02Z

backend/composer/signals.py

+        if target == CSState.EXPORTED and not instance.has_statement_been_exported:
+            instance.has_statement_been_exported = True
+            instance.save()


I think this overlaps with my comment here. My preference would be to use the transition side-effects instead of the signal.

@transition.... def exported(self, *args, **kwargs): self.has_statement_been_exported = True self.save()

backend/composer/models.py

afonsobspinto · 2025-02-14T02:02:59Z

backend/composer/services/export/helpers/export_batch.py

-def do_transition_to_exported_and_get_available_export_batch(
-    export_batch: ExportBatch, user: User
-):
+def filter_statements_with_exported_transition(qs: QuerySet, user: User) -> QuerySet:


This might be a good use case for .iterator

afonsobspinto · 2025-02-14T02:02:59Z

backend/composer/services/export/helpers/export_batch.py

-def do_transition_to_exported_and_get_available_export_batch(
-    export_batch: ExportBatch, user: User
-):
+def filter_statements_with_exported_transition(qs: QuerySet, user: User) -> QuerySet:


This might be a good use case for .iterator

…ity to extract populationset from NeuronDM ID

…-effects instead of the signal

…neurondm function

Feature/sckan-394-ingestion - update ingestion and statement helper to add populationset on ingestion

…tatement and update permission checks

…rom list view and remove deprecated btn from detailed view

…ted_from_this_populationset_incremental_index to PopulationSet

…ex updates

…Admin

… replace Subject URI with compr_uri

…s during deletion in admin

…migration

…generation in export process

…izer

afonsobspinto

Changes requested in the PRs related with 394 were applied in branch https://github.com/MetaCell/sckan-composer/tree/feature/SCKAN-397

SCKAN-396 - Admin interface changes to support the population set - and deprecate state

Feature/sckan-398-export - Extend export to support population set

afonsobspinto · 2025-02-24T18:55:16Z

Included in #423

D-GopalKrishna added 11 commits February 5, 2025 18:31

SCKAN-394 - add populationset and has_statement_been_exported to KS m…

3e91074

…odels

SCKAN-372 - validate populationset change cannot happen for exported KS

bf5b0d1

SCKAN-394 - add population set to admin and make has_statement_been_e…

9d708c5

…xported read only

SCKAN-372 - serializers, urls, and viewsets for populationset

64648cd

SCKAN-372 - migration files for new fields and custom migration for u…

6842f8a

…pdating new field - has_statement_been_exported

SCKAN-394 - test the populationset name validation

9803ce3

SCKAN-372 - utils and state services update

b7e6745

SCKAN-372 - remove the connectivity statement from export-batch if it…

d8c1c36

… doesn't have EXPORT available transition

Merge remote-tracking branch 'origin/develop' into feature/SCKAN-394

e023325

SCKAN-395 - Add PopulationService integration and API endpoints for P…

5ba1a85

…opulationSet

SCAKN_395 - Add population selection to StatementForm and integrate P…

803f5ba

…opulationService

D-GopalKrishna commented Feb 5, 2025

View reviewed changes

backend/composer/models.py Outdated Show resolved Hide resolved

SCKAN-394 - Rename migration function to improve clarity in has_state…

965104d

…ment_been_exported migration

D-GopalKrishna commented Feb 5, 2025

View reviewed changes

backend/composer/services/cs_ingestion/helpers/statement_helper.py Show resolved Hide resolved

D-GopalKrishna commented Feb 5, 2025

View reviewed changes

backend/composer/services/export/helpers/export_batch.py Outdated Show resolved Hide resolved

D-GopalKrishna commented Feb 5, 2025

View reviewed changes

backend/composer/utils.py Outdated Show resolved Hide resolved

SCKAN-394 - fix has_populationset call in statement helper

72fdc7d

afonsobspinto requested changes Feb 6, 2025

View reviewed changes

D-GopalKrishna added 9 commits February 11, 2025 14:40

SCKAN-394 - use django validators for population name check

280c3d9

SCKAN-394 - migration and fix the util to raise error

dc9a72b

SCKAN-394 - Add signal to update has_statement_been_exported on expor…

bb74ad2

…t state change

SCKAN-394 - Add function to get or create neurondm PopulationSet in s…

10f2ddd

…tatement helper

SCKAN-374 - fix the signals and move to post_transition logic

7a0f8e0

SCKAN-394 - Add population_id to SentenceConnectivityStatement serial…

3131162

…izer and refactor population change validation in ConnectivityStatement forms

SCKAN-394 - Update StatementForm to use population_id instead of popu…

6429773

…lation

SCKAN-394 - Refactor post_transition_cs to update has_statement_been_…

32fc5bd

…exported directly on instance

SCKAN-394 - Refactor export services to filter statements before tran…

bb5ea61

…sitioning to exported state

D-GopalKrishna requested a review from afonsobspinto February 13, 2025 03:50

D-GopalKrishna added 2 commits February 13, 2025 11:59

SCKAN-396 - Add DEPRECATED state to CSState and update ConnectivitySt…

6e62a52

…atement constraints

SCKAN-394 - get population in connectivity statement ingestion and ad…

965139c

…d utility to extract population set from statement ID

SCKAN-394 - Update label extraction in for_composer to use lrdf rdfs …

94add82

…label

afonsobspinto requested changes Feb 14, 2025

View reviewed changes

afonsobspinto reviewed Feb 14, 2025

View reviewed changes

D-GopalKrishna and others added 18 commits February 14, 2025 12:12

SCKAN-394 - change sparc-nlp population extraction logic and add util…

d5ed0bc

…ity to extract populationset from NeuronDM ID

SCKAN-394 - move has_statement_been_exported logic to transition side…

1d1ae03

…-effects instead of the signal

SCKAN-394 - optimize the exported transition query by using .iterator

05ff700

Merge branch 'feature/SCKAN-394' into feature/SCKAN-394-ingestion

1a0d3ba

SCKAN-394 - remove debug print statement from get_populationset_from_…

239631c

…neurondm function

Merge pull request #425 from MetaCell/feature/SCKAN-394-ingestion

f8f1537

Feature/sckan-394-ingestion - update ingestion and statement helper to add populationset on ingestion

SCKAN-394 - Implement deprecation process in delete for ConnectivityS…

2bb0fe5

…tatement and update permission checks

SCKAN-394 -add custom delete logic for admin + filter deprecated CS f…

6b143e2

…rom list view and remove deprecated btn from detailed view

SCKAN-398 - Add compr_uri field to ConnectivityStatement and cs_expor…

0739e35

…ted_from_this_populationset_incremental_index to PopulationSet

SCKAN-398 - migration for handling URI generation and incremental ind…

3c2dcec

…ex updates

SCKAN-398 - Add compr_uri to readonly fields in ConnectivityStatement…

613b1ae

…Admin

SCKAN-398 - update export csv - to add a new row for composer_uri and…

25dadd0

… replace Subject URI with compr_uri

SCKAN-394 - Remove unnecessary message handling for deprecated object…

599dc6d

…s during deletion in admin

SCKAN-398 - fix incremental index generation for compr_uri in custom …

0d5fefd

…migration

SCKAN-398 - Ensure incremental index is checked only once during URI …

a160ebd

…generation in export process

SCKAN-398 - update the label in Composer ui for populationset

c13cb92

SCKAN-394 - fix: use CSState for deprecate status in admin

f06cd48

SCKAN-394 - do not return DEPRECATE as available transition in serial…

77e619a

…izer

afonsobspinto previously approved these changes Feb 21, 2025

View reviewed changes

afonsobspinto requested a review from ddelpiano February 21, 2025 17:01

Merge pull request #426 from MetaCell/feature/SCKAN-394-admin

4a2ced4

SCKAN-396 - Admin interface changes to support the population set - and deprecate state

afonsobspinto dismissed their stale review via 4a2ced4 February 21, 2025 19:20

afonsobspinto marked this pull request as draft February 21, 2025 23:39

Merge pull request #429 from MetaCell/feature/SCKAN-398-export

f43c1f8

Feature/sckan-398-export - Extend export to support population set

afonsobspinto closed this Feb 24, 2025

ddelpiano deleted the feature/SCKAN-394 branch July 22, 2025 14:38

Feature/sckan-394-395-396 - Population set - Backend, UI, and Admin UI #420

Feature/sckan-394-395-396 - Population set - Backend, UI, and Admin UI #420

Uh oh!

Conversation

D-GopalKrishna commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

More on the PR:

The Backend changes:

The Composer UI changes:

The Admin UI changes:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

afonsobspinto Feb 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

afonsobspinto Feb 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

afonsobspinto Feb 14, 2025

Choose a reason for hiding this comment

Uh oh!

afonsobspinto Feb 14, 2025

Choose a reason for hiding this comment

Uh oh!

afonsobspinto left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afonsobspinto commented Feb 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

D-GopalKrishna commented Feb 5, 2025 •

edited

Loading

afonsobspinto left a comment •

edited

Loading