Skip to content

Migrate gcs to new system tests design #22778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Apr 28, 2022
Merged

Migrate gcs to new system tests design #22778

merged 18 commits into from
Apr 28, 2022

Conversation

bhirsz
Copy link
Contributor

@bhirsz bhirsz commented Apr 6, 2022

Migrate GCS system tests/example dags to be compliant with new design (AIP-47).

Covers
airflow/providers/google/cloud/example_dags/example_gcs.py
airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
from #22447

@bhirsz bhirsz requested review from mik-laj and turbaszek as code owners April 6, 2022 11:37
@boring-cyborg boring-cyborg bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Apr 6, 2022
@bhirsz
Copy link
Contributor Author

bhirsz commented Apr 6, 2022

The PR is bigger than I want but it's mostly because there are several setup fixtures in old design (in pytest classes) that can now be covered by operators - thus the structure changed and multiple examples had to be done in one go. Everything relates to gcs though.

joppevos
joppevos reviewed Apr 6, 2022
View reviewed changes
@Bowrna
Copy link
Contributor

Bowrna commented Apr 10, 2022

@bhirsz
I tried to write a similar system tests design for Elasticsearch and I was running into issues. (#22811)
Can you help me by telling me if you faced similar issues like the one in the comments in this PR?
It would be helpful if you could check this PR and point out mistakes if there is any in it.

@bhirsz
Copy link
Contributor Author

bhirsz commented Apr 11, 2022

I tried to write a similar system tests design for Elasticsearch and I was running into issues. (#22811)
Can you help me by telling me if you faced similar issues like the one in the comments in #22811 (comment) PR?
It would be helpful if you could check this PR and point out mistakes if there is any in it.

I've commented in your PR. But I've also encountered this problem whenever I run the same DAG_ID twice in the same environment (resetting env or using unique name between runs helps). I will investigate why it happens.

Bartlomiej Hirsz added 2 commits April 12, 2022 06:10
Migrate gcs to new system tests design
e48aae7 
Change-Id: I2aeb518d236296bcdbd489850ed1813cefc0ce3f
Remove unnecessary chain from DAG dependencies
82a00c3 
Change-Id: I2558eab8a98706e693c0cc9bed8bc33354e6a23f
@bhirsz bhirsz mentioned this pull request Apr 12, 2022
Merged
Bartlomiej Hirsz added 2 commits April 12, 2022 08:53
Fix issue with running system tests with not unique DAG ID
4c08fa6 
Change-Id: I4d7d88e13de00b25cbce66b0319870cae1be249b
Migrate GCS sensor examples
5bb09f6 
Change-Id: Iec9be2369a93f2b9bab62f6e5eef298492852c90
@bhirsz bhirsz force-pushed the gcs_system_tests branch from 7665283 to 5bb09f6 Compare April 12, 2022 09:01
@bhirsz
Copy link
Contributor Author

bhirsz commented Apr 13, 2022

I tried to write a similar system tests design for Elasticsearch and I was running into issues. (#22811)
Can you help me by telling me if you faced similar issues like the one in the comments in #22811 (comment) PR?
It would be helpful if you could check this PR and point out mistakes if there is any in it.

I've commented in your PR. But I've also encountered this problem whenever I run the same DAG_ID twice in the same environment (resetting env or using unique name between runs helps). I will investigate why it happens.

Resolved by changing State.NONE (which works only for Tasks) to State.Queued (default state for dag)

bhirsz and others added 7 commits April 13, 2022 07:41
@bhirsz
Merge branch 'main' into gcs_system_tests
a1a685c
Fix system tests failing on importable checks
3bffe04 
Change-Id: I8af21344a836c232ff505c1442b960253eec31b7
Merge branch 'gcs_system_tests' of https://github.com/lwyszomi/airflow …
6492a1e 
…into gcs_system_tests

Change-Id: Ie7a572289872d10c649d8697a5819ddcac9d9e99
@potiuk
Migrate gcs to new system tests design
16edd36 
Change-Id: I2aeb518d236296bcdbd489850ed1813cefc0ce3f
@potiuk
Remove unnecessary chain from DAG dependencies
838f865 
Change-Id: I2558eab8a98706e693c0cc9bed8bc33354e6a23f
@potiuk
Migrate GCS sensor examples
93ea975 
Change-Id: Iec9be2369a93f2b9bab62f6e5eef298492852c90
@potiuk
Fix system tests failing on importable checks
6e8baa6 
Change-Id: I8af21344a836c232ff505c1442b960253eec31b7
@potiuk potiuk force-pushed the gcs_system_tests branch from 6492a1e to 6e8baa6 Compare