WIP: Setting up git-secrets #53

s-rubenstein · 2017-07-26T19:29:45Z

If people have bandwidth, I would love feedback on the README.
Attached is also an example of what the setup script is going to look like, and how to add things to git-secrets. Likely, since we are pushing the git hooks, I suspect the install, moving, and cleaning up isn't going to be necessary, but wanted to show the current state of things. (Also am not yet certain if the install does anything other than create the hooks. If it does, it needs to include that.)

markfickett · 2017-07-27T14:01:20Z

README.md

+
+Download the git-secrets tool.
+Place the tool somewhere in your path, using either:
+* make install


Could you make this part of setup_env.sh?

Maybe something like

if which brew then brew install git-secrets else git clone https://github.com/awslabs/git-secrets.git cd git-secrets make install cd .. rm -rf git-secrets fi

I haven't tried that, but I suspect it would work. I think you still need to download git-secrets before running the brew command, but I can talk to Mohs to see if we can duplicate the effects of a manual install in a script. (To try to do it from a fresh computer.)

If he doesn't have bandwidth I can reset my local environment to try that.

Using their sudo make install I still got permission errors, until I did sudo chmod 755 /usr/local/bin/git-secrets. (I couldn't install without sudo; I got cp: cannot create regular file ‘/usr/local/bin/git-secrets’: Permission denied.)

I think it's OK not to auto-setup, but it'd be useful to at least have instructions for Linux, which I think should go:

git clone https://github.com/awslabs/git-secrets.git cd git-secrets sudo make install sudo chmod 755 /usr/local/bin/git-secrets

and then a "See README for setup instructions if this fails" in setup_git_secrets or wherever is closest to the failure point if you haven't installed it yet.

I'll add the Linux stuff tomorrow, thank you for testing that out for me.

Thanks! Please use the steps in the previous comment (with sudo and chmod) though -- just plain make install doesn't work (it tries to install to a global location w/o sufficient permissions).

markfickett · 2017-07-27T14:03:57Z

tools/setup_git_secrets.sh

@@ -0,0 +1,14 @@
+#!/bin/bash -e


I think since we have a shared hooks directory, we don't actually need this setup script. If you want to put a comment either in the README or in one of the hooks saying the hooks are auto-generated by git secrets --install I think that's good.

The problem is that although the hooks are shared, the results of the git secrets --add commands are stored locally. Probably the installation pieces aren't necessary, but a centralized place to put all the adds would be good.

Oh, I see. Bummer! I think that's a blocker; having to pay attention / do work to keep the secrets list in sync defeats the utility of having something check for secrets, to me. (It's the difference between one-time setup, where if I forget to install git-secrets some scripts break, and having to remember to re-run the installs when we add/change what patterns to looks for; the potential for silent failures.)

On the RDR, we got away with a simple grep. I'm inclined to stick with something like that. It looks like this https://github.com/vanderbilt/pmi-data/blob/master/ci/test.sh#L7 :

echo "Grepping for checked-in credentials..." set +e # OK if grep does not find any matches. KEY_FILES=`grep -ril "BEGIN PRIVATE KEY" . | grep -v $0 | grep -v oauth2client` set -e if [ "${KEY_FILES}" ] then echo "No keys may be checked in, but found: $KEY_FILES" exit 1 fi

I think I agree with you, although theoretically we could add something to the hooks to run the script adding all our secrets before making a commit.

It may make sense to discuss this at, before, or after the standup so we all get on the same page about what our plan is? Or have a separate meeting to discuss it.

markfickett · 2017-07-27T14:04:30Z

tools/setup_git_secrets.sh

+mv hooks/pre-commit .
+mv hooks/prepare-commit-msg .
+rm -rf hooks
+git secrets --add 'private_key_id'


The git-secrets config file should be checked in, right? I don't see it in your PR.

The git-secrets config file is actually just part of your regular git config. workbench/.git/config Despite the fact that we want to push the git secrets configuration, we don't want to push the git config file. (Which is why I explored the idea of a script as a way to solve that?)

Changing ID used for cohorts in external API to be the numeric database ID, assigned by the server on creation.

The nested password args caused problems for macs, hard coded the -p flag with the password argument to avoid that problem, rather do it as one variable.

…#56)

…us/workbench into blrubenstein/git-secrets

markfickett · 2017-08-08T20:28:22Z

README.md

+
+Download the git-secrets tool.
+Place the tool somewhere in your path, using either:
+* make install


Using their sudo make install I still got permission errors, until I did sudo chmod 755 /usr/local/bin/git-secrets. (I couldn't install without sudo; I got cp: cannot create regular file ‘/usr/local/bin/git-secrets’: Permission denied.)

I think it's OK not to auto-setup, but it'd be useful to at least have instructions for Linux, which I think should go:

git clone https://github.com/awslabs/git-secrets.git cd git-secrets sudo make install sudo chmod 755 /usr/local/bin/git-secrets

and then a "See README for setup instructions if this fails" in setup_git_secrets or wherever is closest to the failure point if you haven't installed it yet.

markfickett · 2017-08-08T20:30:02Z

README.md

+* brew install git-secrets
+Run the command from the tools directory:
+```Shell
+./setup_git_secrets


…setup_env.sh

markfickett · 2017-08-09T14:40:43Z

README.md

+Download the git-secrets tool.
+If you have not yet run the setup environment script, run:
+```Shell
+./setup_env


I think several of these are missing tools and .sh. This should be tools/setup_env.sh right?

markfickett · 2017-08-09T14:41:25Z

README.md

+
+git-secrets by default runs every time you make a commit. But if you
+want to manually scan:
+First go to the tools directory and run


I think it's better to skip setup here, since it's just above.

markfickett · 2017-08-09T14:44:37Z

README.md

+
+Download the git-secrets tool.
+Place the tool somewhere in your path, using either:
+* make install


Thanks! Please use the steps in the previous comment (with sudo and chmod) though -- just plain make install doesn't work (it tries to install to a global location w/o sufficient permissions).

markfickett · 2017-08-09T14:45:30Z

README.md

+```Shell
+./setup_env
+```
+Otherwise, if the command fails or you have run it before,


Since you have these commands in setup_env, I don't think you need to include them here too. (It increases risk of them getting out of sync.)

I do agree. That said, I am getting a bit wary of doing so much in the setup env script, since I know that it can fail pretty easily if you have already run one or more steps of it?
That said, there is nothing stopping a developer from going in and looking at the file.

jmandel · 2017-08-09T15:06:46Z

Maybe setup_env.sh should extract code blocks from the README and run them :p

(I joke, but parenthetically: https://en.wikipedia.org/wiki/Literate_programming is a delightful concept.)

jmandel · 2017-08-09T15:23:50Z

README.md

+Download the git-secrets tool.
+Run:
+```Shell
+tools/setup_env.sh


A few issues here.

The first I run setup_env.sh, make install fails because it doesn't have root access. Could fix with sudo, though by default this creates a file in /usr/local/bin that non-root users can't read/execute.

Re-running setup_env.sh, git clone fails with fatal: destination path 'git-secrets' already exists and is not an empty directory.

In trying this out, I noticed that our reference to a pre-publication snapshot of the swagger cli jar has expired.

The tweaks below are blunt, but the result works for me...

diff --git a/tools/download_swagger_cli.sh b/tools/download_swagger_cli.sh index 8f7f547..b8a96e8 100755 --- a/tools/download_swagger_cli.sh +++ b/tools/download_swagger_cli.sh @@ -1 +1 @@ -wget https://oss.sonatype.org/content/repositories/snapshots/io/swagger/swagger-codegen-cli/2.3.0-SNAPSHOT/swagger-codegen-cli-2.3.0-20170716.142514-29.jar -O swagger-codegen-cli.jar +wget https://oss.sonatype.org/content/repositories/snapshots/io/swagger/swagger-codegen-cli/2.3.0-SNAPSHOT/swagger-codegen-cli-2.3.0-20170809.095725-71.jar -O swagger-codegen-cli.jar diff --git a/tools/setup_env.sh b/tools/setup_env.sh index 1cc214c..fd13a46 100755 --- a/tools/setup_env.sh +++ b/tools/setup_env.sh @@ -16,9 +16,10 @@ if which brew then brew install git-secrets else + rm -rf git-secrets git clone https://github.com/awslabs/git-secrets.git cd git-secrets - make install + sudo make install && sudo chmod o+rx /usr/local/bin/git-secrets cd .. rm -rf git-secrets fi

(Note that as we move our dev env to docker, we may want to move angular CLI installation and swagger codegen CLI installation into docker too.)

I was concerned about the issues with re-running setup_env.sh anyway. I'm inclined to just provide installation instructions in the README, and not run the commands in setup_env?

I'm fine with that.

Getting off topic a bit: we should decide what happens on a developer's machine and what happens in docker, in any case.

Installing git and git-secrets feels like a dev host machine concern

Installing angular CLI feels "in the grey area" for me

Installing gradle and maven feels like a "docker dev env" thing to me

I agree that git and git-secrets feels like a dev host machine concern.
The other two questions might warrant asking in the slack or the docker PR?

…setup_env.sh

…us/workbench into blrubenstein/git-secrets

dmohs · 2017-08-09T20:35:58Z

hooks/pre-commit

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+{


Option to possibly provide a nicer error message to someone who doesn't see this in the README.

{ which git-secrets; } || { echo 'git-secrets required and not found. See README for installation instructions.' echo exit 1 }

Brian Rubenstein added 4 commits July 25, 2017 15:53

Added git secrets hooks.

0400737

Git secrets setup script.

38ff015

Format for git secrets call.

ed1318f

Updated the README to provide git-secrets information.

8fc4734

s-rubenstein requested a review from jmandel July 26, 2017 19:29

markfickett reviewed Jul 27, 2017

View reviewed changes

danqrodney and others added 16 commits July 31, 2017 16:39

Removing external ID from cohort DB representation. (#54)

8f103c3

Changing ID used for cohorts in external API to be the numeric database ID, assigned by the server on creation.

Database setup script fix for macs. (#52)

9e73ce8

The nested password args caused problems for macs, hard coded the -p flag with the password argument to avoid that problem, rather do it as one variable.

Add error message if db setup script is run from the wrong directory. (…

4d01046

…#56)

Use real CohortsService. (#51)

f18783f

Added autorun of script and updated script to not reset hooks

0c47370

Testing

3a21117

Hide error if no secrets currently exist.

c475f45

Added hooks successfully calling git-secrets setup script

765e87c

Updated readme

e25f93b

Removed set argument for cleaner logs

86de340

Removed set argument for cleaner logs

aa5e1ac

Merge branch 'blrubenstein/git-secrets' of https://github.com/all-of-…

c3e6596

…us/workbench into blrubenstein/git-secrets

Removed secrets management and assumed onetime run.

874f884

Merge branch 'master' into blrubenstein/git-secrets

5facc1f

Merge branch 'master' into blrubenstein/git-secrets

e10894d

Merge branch 'blrubenstein/git-secrets' of https://github.com/all-of-…

b14cfdb

…us/workbench into blrubenstein/git-secrets

markfickett reviewed Aug 8, 2017

View reviewed changes

Brian Rubenstein added 2 commits August 9, 2017 09:53

Updated README to include instructions for linux, and added setup to …

db61077

…setup_env.sh

Merge branch 'master' into blrubenstein/git-secrets

83d1181

markfickett reviewed Aug 9, 2017

View reviewed changes

Brian Rubenstein added 3 commits August 9, 2017 10:54

Updated secrets to manage secrets on commit

f98f7bb

Updated readme and removed the setup script for git secrets

5cb038a

Fixed bug in readme.

84d6abd

Fixed shell command for setup_env

638ca33

jmandel reviewed Aug 9, 2017

View reviewed changes

Brian Rubenstein and others added 20 commits August 9, 2017 11:44

Removed git secrets from setup_env.sh, and added to README

086539a

Use updated URL for downloading Swagger CLI to fix build (#60)

63ecc71

Added git secrets hooks.

b86bead

Git secrets setup script.

f41125b

Format for git secrets call.

c3034d1

Updated the README to provide git-secrets information.

062d797

Added autorun of script and updated script to not reset hooks

b12d8fb

Testing

55a7d3b

Hide error if no secrets currently exist.

89cf18d

Added hooks successfully calling git-secrets setup script

fb32ed9

Updated readme

c0d2a2b

Removed set argument for cleaner logs

b13c238

Removed secrets management and assumed onetime run.

67f4802

Updated README to include instructions for linux, and added setup to …

29a0929

…setup_env.sh

Updated secrets to manage secrets on commit

f31d249

Updated readme and removed the setup script for git secrets

214fc2f

Fixed bug in readme.

02baba2

Fixed shell command for setup_env

c415cba

Removed git secrets from setup_env.sh, and added to README

63e09a2

Merge branch 'blrubenstein/git-secrets' of https://github.com/all-of-…

1a45778

…us/workbench into blrubenstein/git-secrets

dmohs approved these changes Aug 9, 2017

View reviewed changes

Provided more informative error messages if git secrets not installed

b924a3f

s-rubenstein merged commit 6c762f3 into master Aug 9, 2017

s-rubenstein deleted the blrubenstein/git-secrets branch August 9, 2017 20:41

WIP: Setting up git-secrets #53

WIP: Setting up git-secrets #53

Uh oh!

Conversation

s-rubenstein commented Jul 26, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

s-rubenstein Jul 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

s-rubenstein Jul 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmandel commented Aug 9, 2017

Uh oh!

jmandel Aug 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

s-rubenstein Jul 27, 2017 •

edited

Loading

s-rubenstein Jul 27, 2017 •

edited

Loading

jmandel Aug 9, 2017 •

edited

Loading