Skip to content

Conversation

danieldean
Copy link
Contributor

@danieldean danieldean commented Jul 20, 2025

An addition to the tasks sdk apt client to import the same certificate used for the apiserver by getting the path from the config options so that self-signed certificates get trusted. No additional config options and user wise nothing of note has changed other than a self-signed certificate working without workarounds.

Verification remains on so there is otherwise no security implications.
Closes: #53493

Copy link

boring-cyborg bot commented Jul 20, 2025

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

@danieldean
Copy link
Contributor Author

danieldean commented Jul 20, 2025

Although there are no config changes with this the self-signed certificate does need subject alt names set:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 3650 -nodes \
-subj "CN=<common name>" \
-addext "subjectAltName=DNS:localhost,DNS:airflow-apiserver"

For the certificate to be accepted (this is based on the quick start docker-compose.yaml). As well as the cert being passed to curl using curl --cacert for the apiserver health check plus changing the url from HTTP to HTTPS.

I think this should be documented somewhere but I am unsure where the best place would be. It is supposed to be non-production use but some instances may require the additional security even for this.

@Prab-27
Copy link
Contributor

Prab-27 commented Jul 21, 2025

You can run the pre-commit hook locally—this will automatically fix any static check issues

@potiuk
Copy link
Member

potiuk commented Jul 21, 2025

Also, it would be great to add a unit test covering it. @Subham-KRLX -> you said you can collaborate, so maybe adding a unit test as PR to the PR might be a good idea :)

@danieldean danieldean force-pushed the main branch 2 times, most recently from c857759 to debd475 Compare July 21, 2025 10:41
@Subham-KRLX
Copy link

Subham-KRLX commented Jul 21, 2025

Hi @danieldean I have added a unit test for the new SSL certificate config logic in the API client and confirmed that it passes locally
I have opened a PR with this test .

@danieldean
Copy link
Contributor Author

Hi @danieldean I have added a unit test for the new SSL certificate config logic in the API client and confirmed that it passes locally I have opened a PR with this test .

This appears to be based on a different logic contained within the client to that in this PR?

I think you need to use my branch as a base and make a test based on that if possible.

@Subham-KRLX
Copy link

I have rebased my branch on top of yours and updated the base of my PR accordingly. The unit test now aligns with your changes. Please let me know if you need any adjustments or additional coverage.

@danieldean
Copy link
Contributor Author

danieldean commented Jul 22, 2025

I have rebased my branch on top of yours and updated the base of my PR accordingly. The unit test now aligns with your changes. Please let me know if you need any adjustments or additional coverage.

I am unsure what has happened but there appears to be something wrong with this. You have different code in the client.py and other changes pulling in many approvers.

I have built an image from my branch so I can get on with things. It is here if it is of any use to anyone: https://hub.docker.com/r/danieldean/airflow-test

@Subham-KRLX
Copy link

@danieldean ,Could you please clarify which specific changes or tests you want me to update so everything aligns? I will make the adjustments accordingly.

@Subham-KRLX
Copy link

I see the MyPy checks are failing due to type errors involving direct assignment to ssl.SSLContext methods and some type mismatches in task-sdk/tests/task_sdk/init.py. If you can share how you had prefer the SSL method mocking or type annotations to be adjusted . Just let me know if you want me to try specific fixes or take a look at detailed errors to help move this forward.

@danieldean
Copy link
Contributor Author

I have done some reading on testing and revised my methods. I believe static checks and tests all now pass. Just a short how to on this and all is hopefully done.

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clever :)

@potiuk
Copy link
Member

potiuk commented Jul 26, 2025

Rebased it. I think the mypy error is already fixed in main.

@potiuk potiuk merged commit 6dba101 into apache:main Jul 26, 2025
76 checks passed
Copy link

boring-cyborg bot commented Jul 26, 2025

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@potiuk
Copy link
Member

potiuk commented Jul 26, 2025

🎉

@danieldean
Copy link
Contributor Author

@potiuk excellent, thanks for your help. I will take a look at adding to the documentation over the coming week.

potiuk pushed a commit to potiuk/airflow that referenced this pull request Jul 27, 2025
…d) (apache#53574)

* Add cert used for apiserver to client (fix for self-signed)

* Correct indentation, run pre-commit checks

* Add test for cert load

* Update config description for api ssl_cert

* Use mock instead and check error when non-existant CA is not found
(cherry picked from commit 6dba101)

Co-authored-by: Daniel Dean <[email protected]>
potiuk added a commit that referenced this pull request Jul 27, 2025
…d) (#53574) (#53793)

* Add cert used for apiserver to client (fix for self-signed)

* Correct indentation, run pre-commit checks

* Add test for cert load

* Update config description for api ssl_cert

* Use mock instead and check error when non-existant CA is not found
(cherry picked from commit 6dba101)

Co-authored-by: Daniel Dean <[email protected]>
ferruzzi pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 7, 2025
…3574)

* Add cert used for apiserver to client (fix for self-signed)

* Correct indentation, run pre-commit checks

* Add test for cert load

* Update config description for api ssl_cert

* Use mock instead and check error when non-existant CA is not found
fweilun pushed a commit to fweilun/airflow that referenced this pull request Aug 11, 2025
…3574)

* Add cert used for apiserver to client (fix for self-signed)

* Correct indentation, run pre-commit checks

* Add test for cert load

* Update config description for api ssl_cert

* Use mock instead and check error when non-existant CA is not found
@potiuk potiuk added this to the Airflow 3.0.4 milestone Aug 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unable to use a self-signed certificate due to API failing on verify
4 participants