From 71750a3db27233bf1f0238ce8f10a59c65c0fbc8 Mon Sep 17 00:00:00 2001 From: Simon Massey Date: Thu, 14 Jun 2018 11:05:30 +0100 Subject: [PATCH 1/9] first draft --- RFC/RFC001.md | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 RFC/RFC001.md diff --git a/RFC/RFC001.md b/RFC/RFC001.md new file mode 100644 index 00000000..32ebf64b --- /dev/null +++ b/RFC/RFC001.md @@ -0,0 +1,77 @@ +# RFC 0001 A stable and forwards compatible public key storage format + +**Feature Name:** Stable public key storage + +**Status:** Draft (first parts sketched) + +**Type:** Enhancement + +**Start Date:** 2018-06-14 + +**Author:** Simon Massey + +**Related components:** Core + +**GitHub issues:** + +* #136 GnuPG2 2.2 vs 2.1 conflicts in keybox format + +## Summary + +A new internal public key storage format that avoids forwards compatibility issues between GPG releases. This proposal will keep forwards compatibility with older versions of git-secret. + +## Motivation + +GPG maintains backwards compatibility but not forwards compatibility. Users typically upgrade GPG but do not downgrade. This means that running a new GPG version will typially update the keyring storage files in a way that is not recognised by older versions of GPG. This is normally not a problem of typicaly GPG usage as users typically only upgrade and do not downgrade. It is a problem for git-secret as the keyring storage is committed to git and shared between users. Some uses can be using an older version of GPG that cannot open the upgraded keyring file. + +## Approach + +git-secret will move away from using the keyring format as shared storage of public keys. Instead it will store public keys as seperate files. The export format is stable and forwards compatibile as it is normal for GPG uses to be running different versions and to exchange keys. Any future bugs that effect git-secret's ability to use the files will also effect typical GPG usage. Such bugs are likely to be fixed by the wider GPG community. + +git-secret may need to store and process meta-data about keys to make it efficient to work with keys that are individually stored. It will use the matchine readable ["colon listings format"](https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS) for this purpose. + +It is anticipated that `bash` and `gawk` will be suffient to be able to impliment and use the new shared key storage format. + +## Design + +The storage format will be: + +1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ascii armour rather than binary format is to make debugging of key issues easier. The file name of the key will be the "64 bit keyid" (field 5 of the `colon listings` format) with extension `.pub.gpg` +1. Key meta data will be stored alongside the key file in the `gpg --keyid-format long --with-colons` format. The file name will be the "64 bit keyid" (field 5 of the `colon listings` format) with extension `.gpg.colon` +1. A "keyring cache" will be create at `~/.gitsecret/cache` and this folder will be added to `.gitignore`. At this location a public keyring will be maintained on a per user bases and it won't be shared by users. +1. A tombestone marker file with file name of the "64 bit keyid" with the extension `.killed` will be created by git-secret-killperson to market a user as killed. + +The storage format will be used by git-secret-tell as follows: + +1. Scan the set of `.gpg.colon` to find all currently told identities. If the told identity is in the current list do nothing. +1. If the told identity isn't listed run `gpg --armor --export` against the users `$HOME` keyring. Run `--keyid-format long --with-colons` of the exported key. The result being two new files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` + +The storage format will be used by git-secret-hide as follows: + +1. The list of "64 bit keyid"s who are told will be computed from the `.gpg.colon` files. This will be checked against the list of "64 bit keyid"s in the "keyring cache" at `~/.gitsecret/cache`. Any missing keys are imported into the "keyring cache". It is anticipated that `grep` and `gawk` will be suffient to be perform this calculation. The code then runs as normal using the keyring cache. + +The storage format will be used by git-secret-killperson as follows: + +TBD + +The storage format will be used by git-secret-xxx as follows: + +TBD + +Forwards compatibility with older versions of git-secret will be mantained as follows: + +TBD + +## Drawbacks + +To maintain forward compatibility the approach requires the eixsting logic to kept working for a perod of time... TBD + +## Alternatives + +What other designs have been considered? Unknown. + +What is the impact of not doing this? Team members are locked out of their secrets broken when one user upgrades. This can go undetected until the victim needs the secrets in a hurry for production support. Bad things then happen. + +## Unresolved questions + +What parts of the design are still to be done? See TBD points above. From ab3265d4b058a04a02131f3e43b11b128174199f Mon Sep 17 00:00:00 2001 From: Simon Massey Date: Thu, 14 Jun 2018 11:15:17 +0100 Subject: [PATCH 2/9] typo --- RFC/RFC001.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/RFC001.md b/RFC/RFC001.md index 32ebf64b..5a35db3c 100644 --- a/RFC/RFC001.md +++ b/RFC/RFC001.md @@ -39,7 +39,7 @@ The storage format will be: 1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ascii armour rather than binary format is to make debugging of key issues easier. The file name of the key will be the "64 bit keyid" (field 5 of the `colon listings` format) with extension `.pub.gpg` 1. Key meta data will be stored alongside the key file in the `gpg --keyid-format long --with-colons` format. The file name will be the "64 bit keyid" (field 5 of the `colon listings` format) with extension `.gpg.colon` 1. A "keyring cache" will be create at `~/.gitsecret/cache` and this folder will be added to `.gitignore`. At this location a public keyring will be maintained on a per user bases and it won't be shared by users. -1. A tombestone marker file with file name of the "64 bit keyid" with the extension `.killed` will be created by git-secret-killperson to market a user as killed. +1. A tombestone marker file with file name of the "64 bit keyid" with the extension `.killed` will be created by git-secret-killperson to mark a user as killed. The storage format will be used by git-secret-tell as follows: From ad663a7561faab15ab2cd0953ec99f467456a290 Mon Sep 17 00:00:00 2001 From: Simon Massey Date: Thu, 14 Jun 2018 11:18:07 +0100 Subject: [PATCH 3/9] mention that forward compatibility will be added later --- RFC/RFC001.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/RFC/RFC001.md b/RFC/RFC001.md index 5a35db3c..90b359ef 100644 --- a/RFC/RFC001.md +++ b/RFC/RFC001.md @@ -46,6 +46,8 @@ The storage format will be used by git-secret-tell as follows: 1. Scan the set of `.gpg.colon` to find all currently told identities. If the told identity is in the current list do nothing. 1. If the told identity isn't listed run `gpg --armor --export` against the users `$HOME` keyring. Run `--keyid-format long --with-colons` of the exported key. The result being two new files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` +Note that the steps to ensure that older versions of git-secret know about the newly told user will be outlined below. + The storage format will be used by git-secret-hide as follows: 1. The list of "64 bit keyid"s who are told will be computed from the `.gpg.colon` files. This will be checked against the list of "64 bit keyid"s in the "keyring cache" at `~/.gitsecret/cache`. Any missing keys are imported into the "keyring cache". It is anticipated that `grep` and `gawk` will be suffient to be perform this calculation. The code then runs as normal using the keyring cache. From a7d181190e696e0b9ca8b543aec262c0738c2bcf Mon Sep 17 00:00:00 2001 From: Simon Massey Date: Thu, 14 Jun 2018 22:41:50 +0100 Subject: [PATCH 4/9] updates to the design --- RFC/RFC001.md | 93 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 72 insertions(+), 21 deletions(-) diff --git a/RFC/RFC001.md b/RFC/RFC001.md index 90b359ef..b11b7ce5 100644 --- a/RFC/RFC001.md +++ b/RFC/RFC001.md @@ -22,57 +22,108 @@ A new internal public key storage format that avoids forwards compatibility issu ## Motivation -GPG maintains backwards compatibility but not forwards compatibility. Users typically upgrade GPG but do not downgrade. This means that running a new GPG version will typially update the keyring storage files in a way that is not recognised by older versions of GPG. This is normally not a problem of typicaly GPG usage as users typically only upgrade and do not downgrade. It is a problem for git-secret as the keyring storage is committed to git and shared between users. Some uses can be using an older version of GPG that cannot open the upgraded keyring file. +GPG maintains backwards compatibility but not forwards compatibility. Running a new GPG version can and will upgrade the keyring storage files in a way that is not recognised by older versions of GPG. This is not normally a problem for typical GPG usage. Users only upgrade and rarely downgrade. It is a problem for git-secret as the keyring storage is committed to git and shared between users. Someone using an older version of GPG can no longer open the upgraded keyring file. ## Approach -git-secret will move away from using the keyring format as shared storage of public keys. Instead it will store public keys as seperate files. The export format is stable and forwards compatibile as it is normal for GPG uses to be running different versions and to exchange keys. Any future bugs that effect git-secret's ability to use the files will also effect typical GPG usage. Such bugs are likely to be fixed by the wider GPG community. +git-secret will move away from using the keyring format as shared storage of public keys. Instead it will store public keys as separate files. The export format is stable and forwards compatible. It is normal for GPG users to be running different versions and to exchange keys successfully. Any future bugs that effect git-secret's ability to use the files will affect typical GPG key exchange usage. Such bugs are likely to be fixed by the wider GPG community. -git-secret may need to store and process meta-data about keys to make it efficient to work with keys that are individually stored. It will use the matchine readable ["colon listings format"](https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS) for this purpose. +git-secret may need to store and process meta-data about keys to make it efficient to work with keys that are individually stored. It will use the machine-readable ["colon listings format"](https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS) for this purpose. -It is anticipated that `bash` and `gawk` will be suffient to be able to impliment and use the new shared key storage format. +It is anticipated that `bash` and `gawk` will be sufficient to be able to implement and use the new shared key storage format. ## Design The storage format will be: -1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ascii armour rather than binary format is to make debugging of key issues easier. The file name of the key will be the "64 bit keyid" (field 5 of the `colon listings` format) with extension `.pub.gpg` -1. Key meta data will be stored alongside the key file in the `gpg --keyid-format long --with-colons` format. The file name will be the "64 bit keyid" (field 5 of the `colon listings` format) with extension `.gpg.colon` -1. A "keyring cache" will be create at `~/.gitsecret/cache` and this folder will be added to `.gitignore`. At this location a public keyring will be maintained on a per user bases and it won't be shared by users. -1. A tombestone marker file with file name of the "64 bit keyid" with the extension `.killed` will be created by git-secret-killperson to mark a user as killed. +1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ASCII armour rather than binary format is to make debugging of key issues easier. The filename of the key will be the "64-bit keyid" (field 5 of the `colon listings` format) `.pub.gpg` +1. Key meta data will be stored alongside the key file in the `gpg --keyid-format long --with-colons` format. The file name will be `.gpg.colon` +1. A "keyring cache" will be created at `~/.gitsecret/cache` and this folder will be added to `.gitignore`. At this location, a public keyring will be maintained on a per user bases and it won't be shared by users. +1. A tombstone marker file with `.killed` will be created at `~/.gitsecret/keys` by git-secret-killperson to mark a user as killed. -The storage format will be used by git-secret-tell as follows: +git-secret-tell will: -1. Scan the set of `.gpg.colon` to find all currently told identities. If the told identity is in the current list do nothing. +1. Scan the set of `.gpg.colon` to find all currently told identities. If the told identity is in the list do nothing. 1. If the told identity isn't listed run `gpg --armor --export` against the users `$HOME` keyring. Run `--keyid-format long --with-colons` of the exported key. The result being two new files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` -Note that the steps to ensure that older versions of git-secret know about the newly told user will be outlined below. +Note that the additional steps to ensure that older versions of git-secret know about the newly told user will be outlined below. -The storage format will be used by git-secret-hide as follows: +git-secret-hide will: -1. The list of "64 bit keyid"s who are told will be computed from the `.gpg.colon` files. This will be checked against the list of "64 bit keyid"s in the "keyring cache" at `~/.gitsecret/cache`. Any missing keys are imported into the "keyring cache". It is anticipated that `grep` and `gawk` will be suffient to be perform this calculation. The code then runs as normal using the keyring cache. +1. The list of "64-bit keyid"s who are told will be computed from the `.gpg.colon` files. This will be checked against the list of "64-bit keyid"s in the "keyring cache" at `~/.gitsecret/cache`. Any missing keys are imported into the "keyring cache". It is anticipated that `grep` and `gawk` will be sufficient to perform this calculation. The code then runs as before using the "keyring cache". -The storage format will be used by git-secret-killperson as follows: +git-secret-whoknows will: -TBD +1. The list of identities will be loaded by parsing the `.gpg.colon` files. Note that multiple identities can be listed against each key. -The storage format will be used by git-secret-xxx as follows: +git-secret-usage will: -TBD +1. Document the git-secret-migrate command discussed in the next section. -Forwards compatibility with older versions of git-secret will be mantained as follows: +git-secret-reveal will: -TBD +* Be unchanged. + +git-secret-remove will: + +* Be unchanged. + +git-secret-list will: + +* Be unchanged. + +git-secret-killperson will: + +1. Remove the key from the keyring cache then delete the `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys`. + +git-secret-init will: + +1. Add `~/.gitsecret/cache` into `.gitignore` and do any initialisation work of the public keyring at that location. + +git-secret-clean will: + +* Be unchanged. + +git-secret-changes will: + +1. Show differences the files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` + +git-secret-add will: + +* Be unchanged. + +A new command git-secret-migrate will: + +1. Create the folder `~/.gitsecret/cache` and add it to the `.gitignore` file. +1. Extract any keys from the old keyring generating `.pub.gpg` and `.gpg.colon` files in `~/.gitsecret/keys` + +## Version compatibility + +Backwards compatibility will the old keyring storage approach will be maintained as follows: + +1. For each changed command a guard will be added that checks for the existence of the .gitsecret/cache +1. If the folder exists it proceeds as normal. +1. If it does not exist it will report that the repo was initialised by an older version of git-secret and tell the user to run git-secret-migrate + +Forwards compatibility with older versions of git-secret will be maintained as follows. + +git-secret-hide will: + +1. Have a guard that will check for the existence of the old keyring. If it exists it will check it for any new public keys and extract them into the new format prior to running. + +git-secret-tell will: + +1. Will check for the existence of the old keyring. If it exists it will load the new public key into it. ## Drawbacks -To maintain forward compatibility the approach requires the eixsting logic to kept working for a perod of time... TBD +To maintain forward compatibility the approach requires the existing logic to kept working for a period of time. We can give a deprecated warning if the forwards compatibility logic is running. The warning can be suppressed using a command-line flag. ## Alternatives What other designs have been considered? Unknown. -What is the impact of not doing this? Team members are locked out of their secrets broken when one user upgrades. This can go undetected until the victim needs the secrets in a hurry for production support. Bad things then happen. +What is the impact of not doing this? Team members are locked out of secrets when only one other team member upgrades GPG. This can go undetected until the victim needs the secrets in a hurry for production support. Bad things then happen. ## Unresolved questions From 6a7e0aada1a915176235727890fd3096b1067894 Mon Sep 17 00:00:00 2001 From: Simon Massey Date: Thu, 14 Jun 2018 22:43:08 +0100 Subject: [PATCH 5/9] formatting --- RFC/RFC001.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/RFC/RFC001.md b/RFC/RFC001.md index b11b7ce5..d9ece589 100644 --- a/RFC/RFC001.md +++ b/RFC/RFC001.md @@ -1,4 +1,4 @@ -# RFC 0001 A stable and forwards compatible public key storage format +# RFC 0001 - A stable and forwards compatible public key storage format **Feature Name:** Stable public key storage @@ -97,7 +97,7 @@ A new command git-secret-migrate will: 1. Create the folder `~/.gitsecret/cache` and add it to the `.gitignore` file. 1. Extract any keys from the old keyring generating `.pub.gpg` and `.gpg.colon` files in `~/.gitsecret/keys` -## Version compatibility +## Version Compatibility Backwards compatibility will the old keyring storage approach will be maintained as follows: From 7dc7c0d24addf117781a876c7d171f1743c6c307 Mon Sep 17 00:00:00 2001 From: Simon Massey Date: Thu, 14 Jun 2018 22:43:57 +0100 Subject: [PATCH 6/9] status review --- RFC/RFC001.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC/RFC001.md b/RFC/RFC001.md index d9ece589..479ae6a1 100644 --- a/RFC/RFC001.md +++ b/RFC/RFC001.md @@ -2,7 +2,7 @@ **Feature Name:** Stable public key storage -**Status:** Draft (first parts sketched) +**Status:** Review **Type:** Enhancement From 11278922527190f0a3b2a1c8bd22593a3a8b74c7 Mon Sep 17 00:00:00 2001 From: Simon Massey Date: Thu, 14 Jun 2018 23:08:39 +0100 Subject: [PATCH 7/9] final --- RFC/RFC001.md | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/RFC/RFC001.md b/RFC/RFC001.md index 479ae6a1..86f4f969 100644 --- a/RFC/RFC001.md +++ b/RFC/RFC001.md @@ -2,7 +2,7 @@ **Feature Name:** Stable public key storage -**Status:** Review +**Status:** Final **Type:** Enhancement @@ -22,35 +22,36 @@ A new internal public key storage format that avoids forwards compatibility issu ## Motivation -GPG maintains backwards compatibility but not forwards compatibility. Running a new GPG version can and will upgrade the keyring storage files in a way that is not recognised by older versions of GPG. This is not normally a problem for typical GPG usage. Users only upgrade and rarely downgrade. It is a problem for git-secret as the keyring storage is committed to git and shared between users. Someone using an older version of GPG can no longer open the upgraded keyring file. +GPG maintains backwards compatibility but not forwards compatibility. Running a new GPG version can and will upgrade the keyring storage files in a way that is not recognised by older versions of GPG. This is not normally a problem for typical GPG usage. Users will upgrade and rarely downgrade. It is a problem for git-secret as the keyring storage is committed to git and shared between users. Someone using an older version of GPG can no longer open the upgraded keyring file. ## Approach -git-secret will move away from using the keyring format as shared storage of public keys. Instead it will store public keys as separate files. The export format is stable and forwards compatible. It is normal for GPG users to be running different versions and to exchange keys successfully. Any future bugs that effect git-secret's ability to use the files will affect typical GPG key exchange usage. Such bugs are likely to be fixed by the wider GPG community. +git-secret will move away from using the keyring format as shared storage of public keys. Instead it will store public keys as exported keys in ASCII armor format. The public key export format is stable and forwards compatible. GPG users will typically be running different GPG or PGP versions and are able to exchange keys successfully. Any future bugs that effect git-secret's ability to use the exported public keys will affect typical GPG or PGP key exchange usage. Such bugs are likely to be caught and fixed by the wider opensource community. -git-secret may need to store and process meta-data about keys to make it efficient to work with keys that are individually stored. It will use the machine-readable ["colon listings format"](https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS) for this purpose. +git-secret may need to store and process meta-data about keys to make it efficient to work with keys that are stored witin individual files. It will use the machine-readable ["colon listings format"](https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS) for this purpose. -It is anticipated that `bash` and `gawk` will be sufficient to be able to implement and use the new shared key storage format. +It is anticipated that `bash` and `gawk` will be sufficient to be able to use the new shared key storage format efficiently. ## Design -The storage format will be: +The new storage format will be implimented as follows: -1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ASCII armour rather than binary format is to make debugging of key issues easier. The filename of the key will be the "64-bit keyid" (field 5 of the `colon listings` format) `.pub.gpg` +1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ASCII armour rather than binary format is to make debugging of key issues easier. The filename of the key will `.pub.gpg` (using Field 5 the "64-bit keyid" of the colon listings format) 1. Key meta data will be stored alongside the key file in the `gpg --keyid-format long --with-colons` format. The file name will be `.gpg.colon` -1. A "keyring cache" will be created at `~/.gitsecret/cache` and this folder will be added to `.gitignore`. At this location, a public keyring will be maintained on a per user bases and it won't be shared by users. -1. A tombstone marker file with `.killed` will be created at `~/.gitsecret/keys` by git-secret-killperson to mark a user as killed. +1. A folder `~/.gitsecret/cache` will be added to `.gitignore`. At this location, a public keyring will be maintained on a per user bases and it won't be shared by users. This is simply a "local cache" of the keys used to encrypt files. git-secret-tell will: -1. Scan the set of `.gpg.colon` to find all currently told identities. If the told identity is in the list do nothing. -1. If the told identity isn't listed run `gpg --armor --export` against the users `$HOME` keyring. Run `--keyid-format long --with-colons` of the exported key. The result being two new files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` +1. Scan the set of `*.gpg.colon` files to find all currently told identities. If the told identity is in the list do nothing. +1. If the told identity isn't listed run `gpg --armor --export` against the users `$HOME` keyring. Run `--keyid-format long --with-colons` of the exported key. Then create the two new files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` Note that the additional steps to ensure that older versions of git-secret know about the newly told user will be outlined below. git-secret-hide will: -1. The list of "64-bit keyid"s who are told will be computed from the `.gpg.colon` files. This will be checked against the list of "64-bit keyid"s in the "keyring cache" at `~/.gitsecret/cache`. Any missing keys are imported into the "keyring cache". It is anticipated that `grep` and `gawk` will be sufficient to perform this calculation. The code then runs as before using the "keyring cache". +1. The list of "64-bit keyid"s who are told will be computed from the `*.gpg.colon` files. This will be checked against the list of "64-bit keyid"s in the "keyring cache" at `~/.gitsecret/cache`. Any missing keys are imported into the "keyring cache". It is anticipated that `gawk` will be sufficient to perform this calculation. The code then runs as before using the "keyring cache". + +Note that the additional steps to ensure that older versions of git-secret know about the newly told user will be outlined below. git-secret-whoknows will: @@ -115,6 +116,11 @@ git-secret-tell will: 1. Will check for the existence of the old keyring. If it exists it will load the new public key into it. +git-secret-killperson + +1. Will check for the existence of the old keyring. If it exists it will delete the user from it. + + ## Drawbacks To maintain forward compatibility the approach requires the existing logic to kept working for a period of time. We can give a deprecated warning if the forwards compatibility logic is running. The warning can be suppressed using a command-line flag. @@ -127,4 +133,4 @@ What is the impact of not doing this? Team members are locked out of secrets whe ## Unresolved questions -What parts of the design are still to be done? See TBD points above. +What parts of the design are still to be done? None. From 2cef2a337ff8da8bffb1dd176f2b1e39ff7c11ac Mon Sep 17 00:00:00 2001 From: Simon Massey Date: Thu, 14 Jun 2018 23:15:46 +0100 Subject: [PATCH 8/9] final --- RFC/RFC001.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/RFC/RFC001.md b/RFC/RFC001.md index 86f4f969..e8d44128 100644 --- a/RFC/RFC001.md +++ b/RFC/RFC001.md @@ -28,7 +28,7 @@ GPG maintains backwards compatibility but not forwards compatibility. Running a git-secret will move away from using the keyring format as shared storage of public keys. Instead it will store public keys as exported keys in ASCII armor format. The public key export format is stable and forwards compatible. GPG users will typically be running different GPG or PGP versions and are able to exchange keys successfully. Any future bugs that effect git-secret's ability to use the exported public keys will affect typical GPG or PGP key exchange usage. Such bugs are likely to be caught and fixed by the wider opensource community. -git-secret may need to store and process meta-data about keys to make it efficient to work with keys that are stored witin individual files. It will use the machine-readable ["colon listings format"](https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS) for this purpose. +git-secret may need to store and process meta-data about keys to make it efficient to work with keys that are stored within individual files. It will use the machine-readable ["colon listings format"](https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS) for this purpose. It is anticipated that `bash` and `gawk` will be sufficient to be able to use the new shared key storage format efficiently. @@ -36,14 +36,14 @@ It is anticipated that `bash` and `gawk` will be sufficient to be able to use th The new storage format will be implimented as follows: -1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ASCII armour rather than binary format is to make debugging of key issues easier. The filename of the key will `.pub.gpg` (using Field 5 the "64-bit keyid" of the colon listings format) +1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ASCII armour rather than binary format is to make debugging of key issues easier. The filename of the key will be `.pub.gpg` (using Field 5 the "64-bit keyid" of the colon listings format) 1. Key meta data will be stored alongside the key file in the `gpg --keyid-format long --with-colons` format. The file name will be `.gpg.colon` -1. A folder `~/.gitsecret/cache` will be added to `.gitignore`. At this location, a public keyring will be maintained on a per user bases and it won't be shared by users. This is simply a "local cache" of the keys used to encrypt files. +1. A folder `~/.gitsecret/cache` will be added to `.gitignore`. At this location, a public keyring will be maintained on a per user bases and won't be shared between users. This is simply a "keyring cache" of the keys used to encrypt files. git-secret-tell will: -1. Scan the set of `*.gpg.colon` files to find all currently told identities. If the told identity is in the list do nothing. -1. If the told identity isn't listed run `gpg --armor --export` against the users `$HOME` keyring. Run `--keyid-format long --with-colons` of the exported key. Then create the two new files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` +1. Scan the set of `*.gpg.colon` files to find all currently told identities. If the given identity is in the list do nothing. +1. If the given identity isn't listed run `gpg --armor --export` against the users `$HOME` keyring. Run `--keyid-format long --with-colons` of the exported key. Then create the two new files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` Note that the additional steps to ensure that older versions of git-secret know about the newly told user will be outlined below. @@ -129,7 +129,7 @@ To maintain forward compatibility the approach requires the existing logic to ke What other designs have been considered? Unknown. -What is the impact of not doing this? Team members are locked out of secrets when only one other team member upgrades GPG. This can go undetected until the victim needs the secrets in a hurry for production support. Bad things then happen. +What is the impact of not doing this? Team members are locked out of secrets when only one other team member upgrades GPG. This can go undetected until the victims needs the secrets in a hurry for production support. Bad things then happen. ## Unresolved questions From 2fde6c00685f638f2bc6ed3306688802547175b4 Mon Sep 17 00:00:00 2001 From: Simon Massey Date: Fri, 15 Jun 2018 08:55:33 +0100 Subject: [PATCH 9/9] final final, finally --- RFC/RFC001.md | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/RFC/RFC001.md b/RFC/RFC001.md index e8d44128..4d769ab5 100644 --- a/RFC/RFC001.md +++ b/RFC/RFC001.md @@ -6,12 +6,12 @@ **Type:** Enhancement +**Related components:** Core + **Start Date:** 2018-06-14 **Author:** Simon Massey -**Related components:** Core - **GitHub issues:** * #136 GnuPG2 2.2 vs 2.1 conflicts in keybox format @@ -26,36 +26,40 @@ GPG maintains backwards compatibility but not forwards compatibility. Running a ## Approach -git-secret will move away from using the keyring format as shared storage of public keys. Instead it will store public keys as exported keys in ASCII armor format. The public key export format is stable and forwards compatible. GPG users will typically be running different GPG or PGP versions and are able to exchange keys successfully. Any future bugs that effect git-secret's ability to use the exported public keys will affect typical GPG or PGP key exchange usage. Such bugs are likely to be caught and fixed by the wider opensource community. +git-secret will move away from using the keyring format as shared storage of public keys. Instead, it will store public keys as exported keys in ASCII armor format. The public key export format is stable and forwards compatible. GPG users will typically be running different GPG or PGP versions and are able to exchange keys successfully. Bugs that effect git-secret's ability to use exported public keys will likey affect typical GPG key exchange usage. Such bugs are likely to be caught and fixed by the wider opensource community. git-secret may need to store and process meta-data about keys to make it efficient to work with keys that are stored within individual files. It will use the machine-readable ["colon listings format"](https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob_plain;f=doc/DETAILS) for this purpose. -It is anticipated that `bash` and `gawk` will be sufficient to be able to use the new shared key storage format efficiently. +It is anticipated that `bash` and `gawk` will be sufficient to work efficiently with the new file formats. ## Design The new storage format will be implimented as follows: -1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ASCII armour rather than binary format is to make debugging of key issues easier. The filename of the key will be `.pub.gpg` (using Field 5 the "64-bit keyid" of the colon listings format) -1. Key meta data will be stored alongside the key file in the `gpg --keyid-format long --with-colons` format. The file name will be `.gpg.colon` +1. Keys will be stored in `~/.gitsecret/keys` in `gpg --armor --export` format. The use of ASCII armour rather than binary format is to make debugging of key related issues easier. The filename of the key will be `.pub.gpg` (using Field 5 the "64-bit keyid" of the colon listings format) +1. Key meta data will be stored alongside the key file in the `gpg --keyid-format long --with-colons` format. The file name will be `.pub.cln` 1. A folder `~/.gitsecret/cache` will be added to `.gitignore`. At this location, a public keyring will be maintained on a per user bases and won't be shared between users. This is simply a "keyring cache" of the keys used to encrypt files. git-secret-tell will: -1. Scan the set of `*.gpg.colon` files to find all currently told identities. If the given identity is in the list do nothing. -1. If the given identity isn't listed run `gpg --armor --export` against the users `$HOME` keyring. Run `--keyid-format long --with-colons` of the exported key. Then create the two new files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` +1. Scan the set of `*.pub.cln` files to find all currently told identities. If the given identity is in the list do nothing. +1. If the given identity isn't listed run `gpg --armor --export` against the users `$HOME` keyring to create the `.pub.gpg`. +1. Run `--keyid-format long --with-colons` of the exported key to create the `.pub.cln`. -Note that the additional steps to ensure that older versions of git-secret know about the newly told user will be outlined below. +Note that the additional steps to ensure that older versions of git-secret know about the newly told identity will be outlined below. git-secret-hide will: -1. The list of "64-bit keyid"s who are told will be computed from the `*.gpg.colon` files. This will be checked against the list of "64-bit keyid"s in the "keyring cache" at `~/.gitsecret/cache`. Any missing keys are imported into the "keyring cache". It is anticipated that `gawk` will be sufficient to perform this calculation. The code then runs as before using the "keyring cache". +1. Extract the list of "64-bit keyid"s who are told from the `*.pub.cln` files. Note that multiple identities can be listed against each key. +1. Checked this against the list of "64-bit keyid"s in the "keyring cache" at `~/.gitsecret/cache`. +1. Import any missing keys into the "keyring cache". It is anticipated that `gawk` will be sufficient to perform this calculation. +1. Run the current logic using the "keyring cache". Note that the additional steps to ensure that older versions of git-secret know about the newly told user will be outlined below. git-secret-whoknows will: -1. The list of identities will be loaded by parsing the `.gpg.colon` files. Note that multiple identities can be listed against each key. +1. The list of identities will be loaded by parsing the `.pub.cln` files. Note that multiple identities can be listed against each key. git-secret-usage will: @@ -75,11 +79,13 @@ git-secret-list will: git-secret-killperson will: -1. Remove the key from the keyring cache then delete the `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys`. +1. Remove the key from the keyring cache. +1. Delete both `.pub.gpg` and `.pub.cln` files. git-secret-init will: -1. Add `~/.gitsecret/cache` into `.gitignore` and do any initialisation work of the public keyring at that location. +1. Add `~/.gitsecret/cache` into `.gitignore`. +1. Run any current logic using the ignored "keyring cache". git-secret-clean will: @@ -87,7 +93,7 @@ git-secret-clean will: git-secret-changes will: -1. Show differences the files `.pub.gpg` and `.gpg.colon` in `~/.gitsecret/keys` +1. Show differences the `.pub.gpg` and `.pub.cln` files in `~/.gitsecret/keys`. git-secret-add will: @@ -96,13 +102,13 @@ git-secret-add will: A new command git-secret-migrate will: 1. Create the folder `~/.gitsecret/cache` and add it to the `.gitignore` file. -1. Extract any keys from the old keyring generating `.pub.gpg` and `.gpg.colon` files in `~/.gitsecret/keys` +1. Extract all keys from the old keyring generating `.pub.gpg` and `.pub.cln` files in `~/.gitsecret/keys` ## Version Compatibility Backwards compatibility will the old keyring storage approach will be maintained as follows: -1. For each changed command a guard will be added that checks for the existence of the .gitsecret/cache +1. For each changed command a guard will be added that checks for the existance of `.gitsecret/cache`. 1. If the folder exists it proceeds as normal. 1. If it does not exist it will report that the repo was initialised by an older version of git-secret and tell the user to run git-secret-migrate