-
Notifications
You must be signed in to change notification settings - Fork 25.3k
[ONNX] Support torch.compile(backend="onnxrt", options=OrtBackendOptions(...))
#107973
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/107973
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 23fc2a8 with merge base d4a9963 ( UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
6e7bc42
to
555c074
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LG w/ comments
torch.compile(backend="onnxrt", options=OrtBackendOptions(...))
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 jobs have failed, first few of them are: trunk / macos-12-py3-arm64 / test (default, 3, 3, macos-m1-12) Details for Dev Infra teamRaised by workflow job |
d9afa40
to
026e677
Compare
026e677
to
cfeb9ee
Compare
…ons(...)) This reworks the DORT backend factory function to support the options kwarg of torch.compile, and defines a concrete OrtBackendOptions type that can be used to influence the backend. Caching is also implemented in order to reuse backends with equal options. Wrapping the backend in auto_autograd also becomes an option, which allows `OrtBackend` to always be returned as the callable for torch.compile; wrapping happens internally if opted into (True by default).
cfeb9ee
to
23fc2a8
Compare
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
…ions(...))` (#107973) This reworks the DORT backend factory function to support the options kwarg of torch.compile, and defines a concrete OrtBackendOptions type that can be used to influence the backend. Caching is also implemented in order to reuse backends with equal options. Wrapping the backend in auto_autograd also becomes an option, which allows `OrtBackend` to always be returned as the callable for torch.compile; wrapping happens internally if opted into (True by default). Lastly, expose options for configuring preferred execution providers (will be attempted first), whether or not to attempt to infer an ORT EP from a torch found device in the graph or inputs, and finally the default/fallback EPs. ### Demo The following demo runs `Gelu` through `torch.compile(backend="onnxrt")` using various backend options through a dictionary form and a strongly typed form. It additionally exports the model through both the ONNX TorchScript exporter and the new TorchDynamo exporter. ```python import math import onnx.inliner import onnxruntime import torch import torch.onnx torch.manual_seed(0) class Gelu(torch.nn.Module): def forward(self, x): return x * (0.5 * torch.erf(math.sqrt(0.5) * x) + 1.0) @torch.compile( backend="onnxrt", options={ "preferred_execution_providers": [ "NotARealEP", "CPUExecutionProvider", ], "export_options": torch.onnx.ExportOptions(dynamic_shapes=True), }, ) def dort_gelu(x): return Gelu()(x) ort_session_options = onnxruntime.SessionOptions() ort_session_options.log_severity_level = 0 dort_gelu2 = torch.compile( Gelu(), backend="onnxrt", options=torch.onnx._OrtBackendOptions( preferred_execution_providers=[ "NotARealEP", "CPUExecutionProvider", ], export_options=torch.onnx.ExportOptions(dynamic_shapes=True), ort_session_options=ort_session_options, ), ) x = torch.randn(10) torch.onnx.export(Gelu(), (x,), "gelu_ts.onnx") export_output = torch.onnx.dynamo_export(Gelu(), x) export_output.save("gelu_dynamo.onnx") inlined_model = onnx.inliner.inline_local_functions(export_output.model_proto) onnx.save_model(inlined_model, "gelu_dynamo_inlined.onnx") print("Torch Eager:") print(Gelu()(x)) print("DORT:") print(dort_gelu(x)) print(dort_gelu2(x)) ``` Pull Request resolved: #107973 Approved by: https://github.com/BowenBao
This reworks the DORT backend factory function to support the options kwarg of torch.compile, and defines a concrete OrtBackendOptions type that can be used to influence the backend.
Caching is also implemented in order to reuse backends with equal options.
Wrapping the backend in auto_autograd also becomes an option, which allows
OrtBackend
to always be returned as the callable for torch.compile; wrapping happens internally if opted into (True by default).Lastly, expose options for configuring preferred execution providers (will be attempted first), whether or not to attempt to infer an ORT EP from a torch found device in the graph or inputs, and finally the default/fallback EPs.
Demo
The following demo runs
Gelu
throughtorch.compile(backend="onnxrt")
using various backend options through a dictionary form and a strongly typed form. It additionally exports the model through both the ONNX TorchScript exporter and the new TorchDynamo exporter.cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov