Freemabd1/rw 15 #112
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -28,4 +28,5 @@ node_modules | |
| npm-debug.log | ||
| tools/*.jar | ||
| tools/cloud_sql_proxy | ||
| /api/sa-key.json | ||
| .DS_Store | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| require_relative "utils/common" | ||
| require "io/console" | ||
| require "json" | ||
| require "optparse" | ||
| require "tempfile" | ||
|
|
||
|
|
@@ -58,6 +59,19 @@ def dev_up(args) | |
| common.run_inline %W{docker-compose up -d db} | ||
| common.status "Running database migrations..." | ||
| common.run_inline %W{docker-compose run db-migration} | ||
| common.status "Creating service account for API..." | ||
| iam_account = "[email protected]" | ||
|
There was a problem hiding this comment. I'm thinking we may just want to use [email protected] here, for a couple reasons:
Note that if you use run_with_creds() / get_service_account_creds_file() -- see below -- it will use that service account. There was a problem hiding this comment. we can certainly do that. I don't have access to create a service account under @appspot.gserviceaccount.com. I can currently only create under all-of-us-workbench-test.iam.gserviceaccount.com and @all-of-us-ehr-dev.iam.gserviceaccount.com There was a problem hiding this comment. You don't need to create a service account, it already exists. (It's the SA that our deployed AppEngine app in test uses.) You will need a key for it, but you are an Owner on all-of-us-workbench-test, so you should have permissions to do that in cloud console or on the command line (like this). |
||
| common.run_inline %W{ | ||
| gcloud iam service-accounts keys create sa-key.json --iam-account #{iam_account} | ||
|
There was a problem hiding this comment. So this will use whatever project and account you might already be gcloud configured to (which may not be the right ones.) We should probably use --project all-of-us-workbench-test here (assuming that's relevant for this command?) and maybe --account . Either way, it probably makes sense to try to reuse logic from run_with_creds() or get_service_account_creds_file() in here, as it's doing much the same thing. There was a problem hiding this comment. I guess i'm still a little confused about this and the above comments. Maybe we can discuss over a call? There was a problem hiding this comment. Definitely. Let's chat on Monday. |
||
| } | ||
| at_exit { | ||
| # use JSON library to parse sa-key.json and extract key ID | ||
| keyFile = File.read("sa-key.json") | ||
| key_id = JSON.parse(keyFile)["private_key_id"] | ||
| common.run_inline %W{ | ||
| gcloud iam service-accounts keys delete #{key_id} --iam-account #{iam_account} | ||
| } | ||
| } | ||
| common.status "Starting API. This can take a while. Thoughts on reducing development cycle time" | ||
| common.status "are here:" | ||
| common.status " https://github.com/all-of-us/workbench/blob/master/api/doc/2017/dev-cycle.md" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| package org.pmiops.workbench.api; | ||
|
|
||
| import com.google.cloud.bigquery.*; | ||
| import org.pmiops.workbench.model.Criteria; | ||
| import org.pmiops.workbench.model.CriteriaListResponse; | ||
| import org.springframework.http.ResponseEntity; | ||
| import org.springframework.web.bind.annotation.RestController; | ||
|
|
||
| import java.util.HashMap; | ||
| import java.util.List; | ||
| import java.util.Map; | ||
| import java.util.logging.Logger; | ||
|
|
||
| @RestController | ||
| public class CohortBuilderController implements CohortBuilderApiDelegate { | ||
|
|
||
| private static final Logger log = Logger.getLogger(CohortBuilderController.class.getName()); | ||
|
|
||
| public static final String CRITERIA_QUERY = | ||
| "SELECT id,\n" + | ||
| "type,\n" + | ||
| "code,\n" + | ||
| "name,\n" + | ||
| "est_count,\n" + | ||
| "is_group,\n" + | ||
| "is_selectable,\n" + | ||
| "domain_id\n" + | ||
| "FROM `pmi-drc-api-test.synpuf.%s`\n" + | ||
| "WHERE parent_id = @parentId\n" + | ||
| "order by id asc"; | ||
|
|
||
| @Override | ||
| public ResponseEntity<CriteriaListResponse> getCriteriaByTypeAndParentId(String type, String parentId) { | ||
|
|
||
| QueryResult result = getQueryResult(type, parentId); | ||
|
|
||
| Map<String, Integer> rm = getResultMapper(result); | ||
|
|
||
| CriteriaListResponse criteriaResponse = new CriteriaListResponse(); | ||
| for (List<FieldValue> row : result.iterateAll()) { | ||
| final Criteria criteria = new Criteria(); | ||
| criteria.setId(row.get(rm.get("id")).getLongValue()); | ||
| criteria.setType(row.get(rm.get("type")).getStringValue()); | ||
| criteria.setCode(row.get(rm.get("code")).getStringValue()); | ||
| criteria.setName(row.get(rm.get("name")).getStringValue()); | ||
| criteria.setCount(row.get(rm.get("est_count")).isNull() ? 0 : row.get(rm.get("est_count")).getLongValue()); | ||
| criteria.setGroup(row.get(rm.get("is_group")).getBooleanValue()); | ||
| criteria.setSelectable(row.get(rm.get("is_selectable")).getBooleanValue()); | ||
| criteria.setDomainId(row.get(rm.get("domain_id")).isNull() ? null : row.get(rm.get("domain_id")).getStringValue()); | ||
| criteriaResponse.addItemsItem(criteria); | ||
| } | ||
|
|
||
| return ResponseEntity.ok(criteriaResponse); | ||
| } | ||
|
|
||
| protected QueryResult getQueryResult(String type, String parentId) { | ||
| BigQuery bigquery = | ||
| new BigQueryOptions.DefaultBigqueryFactory().create(BigQueryOptions.getDefaultInstance()); | ||
|
|
||
| QueryRequest queryRequest = | ||
| QueryRequest.newBuilder(getQueryString(type)) | ||
| .addNamedParameter("parentId", QueryParameterValue.int64(new Integer(parentId))) | ||
| .setUseLegacySql(false) | ||
| .build(); | ||
|
|
||
| // Execute the query. | ||
| QueryResponse response = bigquery.query(queryRequest); | ||
|
|
||
| // Wait for the job to finish | ||
| while (!response.jobCompleted()) { | ||
| response = bigquery.getQueryResults(response.getJobId()); | ||
| } | ||
|
|
||
| // Check for errors. | ||
| if (response.hasErrors()) { | ||
| String firstError = ""; | ||
| if (response.getExecutionErrors().size() != 0) { | ||
| firstError = response.getExecutionErrors().get(0).getMessage(); | ||
| } | ||
| throw new RuntimeException(firstError); | ||
| } | ||
|
|
||
| return response.getResult(); | ||
| } | ||
|
|
||
| protected Map<String, Integer> getResultMapper(QueryResult result) { | ||
|
There was a problem hiding this comment. Eventually we'll probably want to move this and some of the code above into some BigQuery utils class. (Fine to leave it here for now.) |
||
| Map<String, Integer> resultMapper = new HashMap<String, Integer>(); | ||
| int i = 0; | ||
| for (Field field : result.getSchema().getFields()) { | ||
| resultMapper.put(field.getName(), i++); | ||
| } | ||
| return resultMapper; | ||
| } | ||
|
|
||
| protected String getQueryString(String type) { | ||
| return String.format(CRITERIA_QUERY, type + "_criteria"); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Which classes required this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So i get this error without that dependency: java.lang.RuntimeException: Google App Engine runtime detected (the environment variable "com.google.appengine.runtime.version" is set), but unable to resolve appengine-sdk classes. For more details see https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/APPENGINE.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. Thanks for the info!