From fcedd34fa1b7d60254f987ea280b20c3607c09eb Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Tue, 13 Feb 2024 10:17:45 -0800 Subject: [PATCH 01/25] Added Jason Haddix to the credits. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 86eeb4f..dedfe10 100644 --- a/README.md +++ b/README.md @@ -453,6 +453,7 @@ The content features a conversation between two individuals discussing various t - _Caleb Sima_ for pushing me over the edge of whether to make this a public project or not. - _Joel Parish_ for super useful input on the project's Github directory structure. - _Jonathan Dunn_ for spectacular work on the soon-to-be-released universal client. +- _Jason Haddix_ for the idea of a stitch (chained Pattern) to filter content using a local model before sending on to a cloud model, i.e., cleaning customer data using `llama2` before sending on to `gpt-4` for analysis. ### Primary contributors From f22c20a540b31dd74adc8b91fa410311df50f569 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Tue, 13 Feb 2024 10:21:39 -0800 Subject: [PATCH 02/25] Added Joseph Thacker to the credits. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index dedfe10..b865bc7 100644 --- a/README.md +++ b/README.md @@ -453,6 +453,7 @@ The content features a conversation between two individuals discussing various t - _Caleb Sima_ for pushing me over the edge of whether to make this a public project or not. - _Joel Parish_ for super useful input on the project's Github directory structure. - _Jonathan Dunn_ for spectacular work on the soon-to-be-released universal client. +- _Joseph Thacker_ for the idea of a `-c` context flag that adds pre-created context in the `./config/fabric/` directory to all Pattern queries. - _Jason Haddix_ for the idea of a stitch (chained Pattern) to filter content using a local model before sending on to a cloud model, i.e., cleaning customer data using `llama2` before sending on to `gpt-4` for analysis. ### Primary contributors From f8b2f3aab92b4ce76b2c1f2603c77bf600b91bcd Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 07:33:21 -0800 Subject: [PATCH 03/25] Updated analyze_prose_json. --- patterns/analyze_prose_json/system.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/patterns/analyze_prose_json/system.md b/patterns/analyze_prose_json/system.md index 5ea6e76..db2ae82 100644 --- a/patterns/analyze_prose_json/system.md +++ b/patterns/analyze_prose_json/system.md @@ -30,7 +30,7 @@ Common examples that meet this criteria: - Minor expansion on existing ideas, but in a way that's useful. -"C - Incremental" -- Useful expansion or improvement of existing ideas, or a useful description of the past, but no expansion or creation of new ideas. Imagine a novelty score between 50% and 80% for this tier. +"C - Incremental" -- Useful expansion or significant improvement of existing ideas, or a somewhat insightful description of the past, but no expansion on, or creation of, new ideas. Imagine a novelty score between 50% and 80% for this tier. Common examples that meet this criteria: @@ -41,13 +41,15 @@ Common examples that meet this criteria: Common examples that meet this criteria: -- Contains ideas or facts, but they're not new in any way. +- Rehashes of well-known ideas without any new takes or expansions of ideas. +- Contains ideas or facts, but they're not new or improved in any significant way. "F - Stale" -- No new ideas whatsoever. Imagine a novelty score below 20% for this tier. Common examples that meet this criteria: -- Random ramblings that say nothing new. +- Completely trite and unoriginal ideas. +- Heavily cliche or standard ideas. 4. Evaluate the CLARITY of the writing on the following scale. From 5a928525f397e5cf98861520ff41561c924ec99f Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 07:35:39 -0800 Subject: [PATCH 04/25] Updated analyze_prose_json. --- patterns/analyze_prose_json/system.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/patterns/analyze_prose_json/system.md b/patterns/analyze_prose_json/system.md index db2ae82..5743766 100644 --- a/patterns/analyze_prose_json/system.md +++ b/patterns/analyze_prose_json/system.md @@ -34,13 +34,15 @@ Common examples that meet this criteria: Common examples that meet this criteria: -- Valuable collections of resources -- Descriptions of the past with offered observations and takeaways +- Useful collections of resources. +- Descriptions of the past with offered observations and takeaways. +- Minor expansions on existing ideas. "D - Derivative" -- Largely derivative of well-known ideas. Imagine a novelty score between in the 20% to 50% range for this tier. Common examples that meet this criteria: +- Restatement of common knowledge or best practices. - Rehashes of well-known ideas without any new takes or expansions of ideas. - Contains ideas or facts, but they're not new or improved in any significant way. From aa26deef73f11e650039e2c1270919fb3d3250b4 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 10:38:07 -0800 Subject: [PATCH 05/25] New value rating pattern. --- patterns/rate_value/system.md | 52 +++++++++++++++++++++++++++++++++++ patterns/rate_value/user.md | 0 2 files changed, 52 insertions(+) create mode 100644 patterns/rate_value/system.md create mode 100644 patterns/rate_value/user.md diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md new file mode 100644 index 0000000..b0187e5 --- /dev/null +++ b/patterns/rate_value/system.md @@ -0,0 +1,52 @@ +# IDENTITY and PURPOSE + +You are an expert parser and rater of value in text content. Your goal is to determine how much value a reader/listener is receiving per minute of content. + +Take a deep breath and think step-by-step about how best to achieve the best outcome. + +# Steps + +- Fully read and understand the content and what it's trying to communicate and accomplish. + +- Based on the amount of text and the type of content, figure out how long this should take to say/deliver if it was being spoken at normal conversational speed. + +- Extract all instances of value being provided within the content. Value is defined as: + +-- Surprising or novel ideas or revelations. +-- A giveaway of something useful or valuable. +-- Untold and interesting stories. +-- Secret knowledge. +-- Exclusive content. +-- Positive and/or excited reactions to any content delivered. + +- Based on the number of instances of value and the duration of the content, calculate a metric called Value Per Minute (VPS). + +-- Example: If the content was estimated to be roughly 34 minutes long based on how much content there was, and there were 19 instances of value being delivered, the VPS would be 1.79 (34/19) + +# OUTPUT INSTRUCTIONS + +- Output a valid JSON file with the following fields: + +{ + estimated-content-minutes: "(The estimated length of the content based on how much content thee was combined with the type of content and the speed of human speech.)"; + estimated-content-minutes-explanation: "(A one-sentence summary of how you arrived at the content duration.)"; + value-instance-count: "(The number of value instances in the content.)", + vps: "(the calculated VPS score.)", + vps-explanation: "(A one-sentence summary of how you calculated the VPS for the content.)", +} + +EXAMPLE: + +{ + estimated-content-minutes: "34"; + estimated-content-minutes-explanation: "This was a conversation between two people going back and forth, and this is a natural duration given the length of the text provided."; + value-instance-count: "19", + vps: "1.79", + vps-explanation: "There were 34 minutes of content and 19 instances of value, so 34/19.", +} + + + +# INPUT: + +INPUT: diff --git a/patterns/rate_value/user.md b/patterns/rate_value/user.md new file mode 100644 index 0000000..e69de29 From 000acfd59b53f63e1ede0b4be410b3c7552c2bf1 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 10:43:28 -0800 Subject: [PATCH 06/25] Updated rate_value. --- patterns/rate_value/system.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index b0187e5..6b2ed11 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -30,7 +30,7 @@ Take a deep breath and think step-by-step about how best to achieve the best out { estimated-content-minutes: "(The estimated length of the content based on how much content thee was combined with the type of content and the speed of human speech.)"; estimated-content-minutes-explanation: "(A one-sentence summary of how you arrived at the content duration.)"; - value-instance-count: "(The number of value instances in the content.)", + value-instances: "(list of value instances)", vps: "(the calculated VPS score.)", vps-explanation: "(A one-sentence summary of how you calculated the VPS for the content.)", } @@ -40,6 +40,7 @@ EXAMPLE: { estimated-content-minutes: "34"; estimated-content-minutes-explanation: "This was a conversation between two people going back and forth, and this is a natural duration given the length of the text provided."; + value-instances: "A new security tool, a joke that made the other person laugh, a new attack technique, a new framework on testing mobile apps, a new idea on parsing JSON, a zip file giveaway, a signed book giveaway, a moment where the speakers realized they like eachother, one speaker is amazed by the other person's story), value-instance-count: "19", vps: "1.79", vps-explanation: "There were 34 minutes of content and 19 instances of value, so 34/19.", From 36fd6c632f12372f48113850510486c73df4fc64 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 10:46:44 -0800 Subject: [PATCH 07/25] Updated rate_value with credits in the README.md. --- patterns/rate_value/README.md | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 patterns/rate_value/README.md diff --git a/patterns/rate_value/README.md b/patterns/rate_value/README.md new file mode 100644 index 0000000..100a82b --- /dev/null +++ b/patterns/rate_value/README.md @@ -0,0 +1,3 @@ +# Credit + +Co-created by Daniel Miessler and Jason Haddix based on influences from Claude Shannon's Information Theory and Mr. Beast's highly viral content. From 3a4b22bffb4577fc53cb7618a988b057271cdc27 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 13:15:56 -0800 Subject: [PATCH 08/25] Updated rate_value. --- patterns/rate_value/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/rate_value/README.md b/patterns/rate_value/README.md index 100a82b..4267b54 100644 --- a/patterns/rate_value/README.md +++ b/patterns/rate_value/README.md @@ -1,3 +1,3 @@ # Credit -Co-created by Daniel Miessler and Jason Haddix based on influences from Claude Shannon's Information Theory and Mr. Beast's highly viral content. +Co-created by Daniel Miessler and Jason Haddix based on influences from Claude Shannon's Information Theory and Mr. Beast's insanely viral content techniques. From 959987165f1817e768d169ab5430c05db35711a1 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 13:20:21 -0800 Subject: [PATCH 09/25] Updated main readme. --- README.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/README.md b/README.md index b865bc7..717c0c6 100644 --- a/README.md +++ b/README.md @@ -23,9 +23,6 @@ -> [!NOTE] -> We are currently working on some client installation optimizations. If the instructions don't work, check back in a day or two and it should be sorted, and be even easier than before. - ## Navigation - [What and Why](#what-and-why) From 08887de5bb423982d29f951bcec98a1f4c829f30 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:02:03 -0800 Subject: [PATCH 10/25] Updated rate_value. --- patterns/rate_value/system.md | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 6b2ed11..3ab49fc 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -1,28 +1,39 @@ # IDENTITY and PURPOSE -You are an expert parser and rater of value in text content. Your goal is to determine how much value a reader/listener is receiving per minute of content. +You are an expert parser and rater of value in content. Your goal is to determine how much value a reader/listener is being provided in a given piece of content. -Take a deep breath and think step-by-step about how best to achieve the best outcome. +Take a deep breath and think step-by-step about how best to achieve the best outcome using the STEPS below. -# Steps +# STEPS - Fully read and understand the content and what it's trying to communicate and accomplish. -- Based on the amount of text and the type of content, figure out how long this should take to say/deliver if it was being spoken at normal conversational speed. +- Based on the amount of content provided as input, determine how long it should take to say/deliver the content if it was being spoken at normal, English-speaking conversational speed. -- Extract all instances of value being provided within the content. Value is defined as: +- Extract all Instances Of Value being provided within the content. Instances Of Value are defined as: --- Surprising or novel ideas or revelations. --- A giveaway of something useful or valuable. --- Untold and interesting stories. +-- Highly surprising ideas or revelations. +-- A giveaway of something useful or valuable to the audience. +-- Untold and interesting stories with valuable takeaways. -- Secret knowledge. --- Exclusive content. --- Positive and/or excited reactions to any content delivered. +-- Exclusive content that's never been revealed before. +-- Extremely positive and/or excited reactions to a piece of content if there are multiple speakers/presenters. -- Based on the number of instances of value and the duration of the content, calculate a metric called Value Per Minute (VPS). +- Use the following scale of value for each potential Value Instance: + +1 — Vapid +2 - Weak +3 - Average +4 - Notable +5 - Remarkable + +- In order for a given piece of content to register as a Value Instance, it must score at least 4 out of 5 (Notable (4) or Remarkable (5)) on the value scale above. + +- Based on the number of valid (4 or above) instances of value and the duration of the content, calculate a metric called Value Per Minute (VPS). -- Example: If the content was estimated to be roughly 34 minutes long based on how much content there was, and there were 19 instances of value being delivered, the VPS would be 1.79 (34/19) + # OUTPUT INSTRUCTIONS - Output a valid JSON file with the following fields: @@ -47,7 +58,6 @@ EXAMPLE: } - # INPUT: INPUT: From a5bfccdc505932614f7e78737998640b69a17d52 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:08:54 -0800 Subject: [PATCH 11/25] Updated rate_value. --- patterns/rate_value/system.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 3ab49fc..9534431 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -8,7 +8,9 @@ Take a deep breath and think step-by-step about how best to achieve the best out - Fully read and understand the content and what it's trying to communicate and accomplish. -- Based on the amount of content provided as input, determine how long it should take to say/deliver the content if it was being spoken at normal, English-speaking conversational speed. +- Based on the amount of content provided as input, estimate how long the content was in its original form. E.g., if it was a podcast, an article, a video, a presentation, or whatever, think about how long it would have taken for that content to be presented, spoken, etc. Remember that number as the variable "estimated-content-minutes". + +Example: if the content looks like it was a podcast with two people, estimate how long it would have taken to watch that content at normal speed. - Extract all Instances Of Value being provided within the content. Instances Of Value are defined as: From a42a4d7098d7d509a71b9c9efef499b47a132f0a Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:10:42 -0800 Subject: [PATCH 12/25] Updated rate_value. --- patterns/rate_value/system.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 9534431..18f7356 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -8,7 +8,7 @@ Take a deep breath and think step-by-step about how best to achieve the best out - Fully read and understand the content and what it's trying to communicate and accomplish. -- Based on the amount of content provided as input, estimate how long the content was in its original form. E.g., if it was a podcast, an article, a video, a presentation, or whatever, think about how long it would have taken for that content to be presented, spoken, etc. Remember that number as the variable "estimated-content-minutes". +- Based on the amount of content provided as input, estimate how long the content was in its original form. E.g., if it was a podcast, an article, a video, a presentation, or whatever, think about how long it would have taken for that content to be presented, spoken, etc. Remember that number as the variable "estimated-content-minutes". Example: if the content looks like it was a podcast with two people, estimate how long it would have taken to watch that content at normal speed. @@ -39,6 +39,7 @@ Example: if the content looks like it was a podcast with two people, estimate ho # OUTPUT INSTRUCTIONS - Output a valid JSON file with the following fields: +- Remember to take into account multiple speakers speaking simultaneously when calculating estimated-content-minutes, which might mean the content would take less time to complete. { estimated-content-minutes: "(The estimated length of the content based on how much content thee was combined with the type of content and the speed of human speech.)"; From dbc8077e6439e11e92be108614cea07ebe286854 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:16:20 -0800 Subject: [PATCH 13/25] Updated rate_value. --- patterns/rate_value/system.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 18f7356..0941de5 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -8,9 +8,13 @@ Take a deep breath and think step-by-step about how best to achieve the best out - Fully read and understand the content and what it's trying to communicate and accomplish. -- Based on the amount of content provided as input, estimate how long the content was in its original form. E.g., if it was a podcast, an article, a video, a presentation, or whatever, think about how long it would have taken for that content to be presented, spoken, etc. Remember that number as the variable "estimated-content-minutes". +- Estimate the duration of the content if it were to be consumed naturally, using the algorithm below: -Example: if the content looks like it was a podcast with two people, estimate how long it would have taken to watch that content at normal speed. +1. Count the total number of words in the provided transcript. +2. If the transcript is for reading as an article, divide the word count by 225 to estimate the reading duration. +3. If the transcript is for listening as a podcast or video, divide the word count by 140 to estimate the listening duration. +4. Round the calculated duration to the nearest minute. +5. Store that value as estimated-content-minutes. - Extract all Instances Of Value being provided within the content. Instances Of Value are defined as: From bb08ec5ce302922896e18dd225a19a7f83b0e998 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:18:25 -0800 Subject: [PATCH 14/25] Updated rate_value. --- patterns/rate_value/system.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 0941de5..6777616 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -33,9 +33,9 @@ Take a deep breath and think step-by-step about how best to achieve the best out 4 - Notable 5 - Remarkable -- In order for a given piece of content to register as a Value Instance, it must score at least 4 out of 5 (Notable (4) or Remarkable (5)) on the value scale above. +- In order for a given piece of content to register as a valid Value Instance, it must hit the maximum level of value on the scale of value above. So it must score a 5/5 in value quality. -- Based on the number of valid (4 or above) instances of value and the duration of the content, calculate a metric called Value Per Minute (VPS). +- Based on the number of valid (5/5) instances of value and the duration of the content, calculate a metric called Value Per Minute (VPS). -- Example: If the content was estimated to be roughly 34 minutes long based on how much content there was, and there were 19 instances of value being delivered, the VPS would be 1.79 (34/19) From c205e3afa7eb9af6b7b0254c950878f853ac112b Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:23:47 -0800 Subject: [PATCH 15/25] Updated rate_value. --- patterns/rate_value/system.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 6777616..ece0cb5 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -11,8 +11,8 @@ Take a deep breath and think step-by-step about how best to achieve the best out - Estimate the duration of the content if it were to be consumed naturally, using the algorithm below: 1. Count the total number of words in the provided transcript. -2. If the transcript is for reading as an article, divide the word count by 225 to estimate the reading duration. -3. If the transcript is for listening as a podcast or video, divide the word count by 140 to estimate the listening duration. +2. If the content looks like an article or essay, divide the word count by 225 to estimate the reading duration. +3. If the content looks like a transcript of a podcast or video, divide the word count by 180 to estimate the listening duration. 4. Round the calculated duration to the nearest minute. 5. Store that value as estimated-content-minutes. @@ -35,6 +35,17 @@ Take a deep breath and think step-by-step about how best to achieve the best out - In order for a given piece of content to register as a valid Value Instance, it must hit the maximum level of value on the scale of value above. So it must score a 5/5 in value quality. +- In order for a given piece of content to register as a valid Value Instance, it must also relate to one or more of the following topics: + +- The improvement of human flourishing +- Applying AI to human problems +- Life improvement using AI +- New ideas related to human flourishing +- New mental models +- New ways of thinking about the world +- New frameworks for solving problems +- New tools for solving problems + - Based on the number of valid (5/5) instances of value and the duration of the content, calculate a metric called Value Per Minute (VPS). -- Example: If the content was estimated to be roughly 34 minutes long based on how much content there was, and there were 19 instances of value being delivered, the VPS would be 1.79 (34/19) From adb391628ea3304c1dce4e9710183a912789fc4b Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:27:51 -0800 Subject: [PATCH 16/25] Updated rate_value. --- patterns/rate_value/system.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index ece0cb5..8b002ce 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -33,9 +33,11 @@ Take a deep breath and think step-by-step about how best to achieve the best out 4 - Notable 5 - Remarkable -- In order for a given piece of content to register as a valid Value Instance, it must hit the maximum level of value on the scale of value above. So it must score a 5/5 in value quality. +- In order for a given piece of content to register as a valid Value Instance, it must satisfy both of the following criteria: -- In order for a given piece of content to register as a valid Value Instance, it must also relate to one or more of the following topics: +1. Hit the maximum level of value on the scale of value above. So it must score a 5/5 in value quality. + +2. It must relate to one or more of the following topics: - The improvement of human flourishing - Applying AI to human problems @@ -46,15 +48,16 @@ Take a deep breath and think step-by-step about how best to achieve the best out - New frameworks for solving problems - New tools for solving problems -- Based on the number of valid (5/5) instances of value and the duration of the content, calculate a metric called Value Per Minute (VPS). +- Based on the number of valid Instances Of Value and the duration of the content, calculate a metric called Value Per Minute (VPS). + +-- Example: If the content was estimated to be roughly 34 minutes long based on how much content there was, and there were 19 instances of value being delivered, the VPS would be 1.79 (34/19) -- Example: If the content was estimated to be roughly 34 minutes long based on how much content there was, and there were 19 instances of value being delivered, the VPS would be 1.79 (34/19) # OUTPUT INSTRUCTIONS -- Output a valid JSON file with the following fields: -- Remember to take into account multiple speakers speaking simultaneously when calculating estimated-content-minutes, which might mean the content would take less time to complete. +- Output a valid JSON file with the following fields for the input provided. { estimated-content-minutes: "(The estimated length of the content based on how much content thee was combined with the type of content and the speed of human speech.)"; @@ -64,7 +67,7 @@ Take a deep breath and think step-by-step about how best to achieve the best out vps-explanation: "(A one-sentence summary of how you calculated the VPS for the content.)", } -EXAMPLE: +EXAMPLE OUTPUT: { estimated-content-minutes: "34"; From 09bf9d56bacaf7f807c38cef5a75408045e8c9a4 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:29:07 -0800 Subject: [PATCH 17/25] Updated rate_value. --- patterns/rate_value/system.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 8b002ce..4231b03 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -67,17 +67,6 @@ Take a deep breath and think step-by-step about how best to achieve the best out vps-explanation: "(A one-sentence summary of how you calculated the VPS for the content.)", } -EXAMPLE OUTPUT: - -{ - estimated-content-minutes: "34"; - estimated-content-minutes-explanation: "This was a conversation between two people going back and forth, and this is a natural duration given the length of the text provided."; - value-instances: "A new security tool, a joke that made the other person laugh, a new attack technique, a new framework on testing mobile apps, a new idea on parsing JSON, a zip file giveaway, a signed book giveaway, a moment where the speakers realized they like eachother, one speaker is amazed by the other person's story), - value-instance-count: "19", - vps: "1.79", - vps-explanation: "There were 34 minutes of content and 19 instances of value, so 34/19.", -} - # INPUT: From f88a3cd11268af4e88a92d91d43049583cd8db5a Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:33:32 -0800 Subject: [PATCH 18/25] Updated rate_value. --- patterns/rate_value/system.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 4231b03..316436a 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -35,11 +35,14 @@ Take a deep breath and think step-by-step about how best to achieve the best out - In order for a given piece of content to register as a valid Value Instance, it must satisfy both of the following criteria: -1. Hit the maximum level of value on the scale of value above. So it must score a 5/5 in value quality. +1. Hit the "Notable" level of value on the scale of value above. So it must score a 4/5 in value quality. -2. It must relate to one or more of the following topics: +2. It must ALSO relate to one or more of the following topics: - The improvement of human flourishing +- Human creativity +- Human productivity +- Human health - Applying AI to human problems - Life improvement using AI - New ideas related to human flourishing @@ -48,7 +51,7 @@ Take a deep breath and think step-by-step about how best to achieve the best out - New frameworks for solving problems - New tools for solving problems -- Based on the number of valid Instances Of Value and the duration of the content, calculate a metric called Value Per Minute (VPS). +- Based on the number of valid Instances Of Value and the duration of the content (both above 4/5 and also related to those topics above), calculate a metric called Value Per Minute (VPS). -- Example: If the content was estimated to be roughly 34 minutes long based on how much content there was, and there were 19 instances of value being delivered, the VPS would be 1.79 (34/19) From 9bbfa6105bbc8d2e5fe95c44b55de8e29516e230 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:35:46 -0800 Subject: [PATCH 19/25] Updated rate_value. --- patterns/rate_value/system.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 316436a..50dceaa 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -33,9 +33,7 @@ Take a deep breath and think step-by-step about how best to achieve the best out 4 - Notable 5 - Remarkable -- In order for a given piece of content to register as a valid Value Instance, it must satisfy both of the following criteria: - -1. Hit the "Notable" level of value on the scale of value above. So it must score a 4/5 in value quality. +- In order for a given piece of content to register as a valid Value Instance, it must reach or exceed the "Notable" level of value on the scale of value above. So it must score a 4/5 or a 5/5 in value quality. 2. It must ALSO relate to one or more of the following topics: From 99128a9ac58b66777d67461576ac1afaf5be7f7c Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:38:54 -0800 Subject: [PATCH 20/25] Updated rate_value. --- patterns/rate_value/system.md | 28 ++++------------------------ 1 file changed, 4 insertions(+), 24 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index 50dceaa..cab3e2c 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -33,39 +33,19 @@ Take a deep breath and think step-by-step about how best to achieve the best out 4 - Notable 5 - Remarkable -- In order for a given piece of content to register as a valid Value Instance, it must reach or exceed the "Notable" level of value on the scale of value above. So it must score a 4/5 or a 5/5 in value quality. - -2. It must ALSO relate to one or more of the following topics: - -- The improvement of human flourishing -- Human creativity -- Human productivity -- Human health -- Applying AI to human problems -- Life improvement using AI -- New ideas related to human flourishing -- New mental models -- New ways of thinking about the world -- New frameworks for solving problems -- New tools for solving problems +- In order for a given piece of content to register as a valid Value Instance, it must reach or exceed the "Notable" (4/5) level of value on the scale of value above. - Based on the number of valid Instances Of Value and the duration of the content (both above 4/5 and also related to those topics above), calculate a metric called Value Per Minute (VPS). --- Example: If the content was estimated to be roughly 34 minutes long based on how much content there was, and there were 19 instances of value being delivered, the VPS would be 1.79 (34/19) - --- Example: If the content was estimated to be roughly 34 minutes long based on how much content there was, and there were 19 instances of value being delivered, the VPS would be 1.79 (34/19) - - # OUTPUT INSTRUCTIONS - Output a valid JSON file with the following fields for the input provided. { - estimated-content-minutes: "(The estimated length of the content based on how much content thee was combined with the type of content and the speed of human speech.)"; - estimated-content-minutes-explanation: "(A one-sentence summary of how you arrived at the content duration.)"; - value-instances: "(list of value instances)", + estimated-content-minutes: "(estimated-content-minutes)"; + value-instances: "(list of valid value instances)", vps: "(the calculated VPS score.)", - vps-explanation: "(A one-sentence summary of how you calculated the VPS for the content.)", + vps-explanation: "(A one-sentence summary of less than 20 words on how you calculated the VPS for the content.)", } From 225e5031bfa4a8edeb519bba57b1a75582f1c2a9 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 14:42:37 -0800 Subject: [PATCH 21/25] Updated rate_value. --- patterns/rate_value/system.md | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/patterns/rate_value/system.md b/patterns/rate_value/system.md index cab3e2c..6d842c7 100644 --- a/patterns/rate_value/system.md +++ b/patterns/rate_value/system.md @@ -1,6 +1,6 @@ # IDENTITY and PURPOSE -You are an expert parser and rater of value in content. Your goal is to determine how much value a reader/listener is being provided in a given piece of content. +You are an expert parser and rater of value in content. Your goal is to determine how much value a reader/listener is being provided in a given piece of content as measured by a new metric called Value Per Minute (VPM). Take a deep breath and think step-by-step about how best to achieve the best outcome using the STEPS below. @@ -21,21 +21,12 @@ Take a deep breath and think step-by-step about how best to achieve the best out -- Highly surprising ideas or revelations. -- A giveaway of something useful or valuable to the audience. -- Untold and interesting stories with valuable takeaways. --- Secret knowledge. +-- Sharing of an uncommonly valuable resource. +-- Sharing of secret knowledge. -- Exclusive content that's never been revealed before. -- Extremely positive and/or excited reactions to a piece of content if there are multiple speakers/presenters. -- Use the following scale of value for each potential Value Instance: - -1 — Vapid -2 - Weak -3 - Average -4 - Notable -5 - Remarkable - -- In order for a given piece of content to register as a valid Value Instance, it must reach or exceed the "Notable" (4/5) level of value on the scale of value above. - -- Based on the number of valid Instances Of Value and the duration of the content (both above 4/5 and also related to those topics above), calculate a metric called Value Per Minute (VPS). +- Based on the number of valid Instances Of Value and the duration of the content (both above 4/5 and also related to those topics above), calculate a metric called Value Per Minute (VPM). # OUTPUT INSTRUCTIONS @@ -44,8 +35,8 @@ Take a deep breath and think step-by-step about how best to achieve the best out { estimated-content-minutes: "(estimated-content-minutes)"; value-instances: "(list of valid value instances)", - vps: "(the calculated VPS score.)", - vps-explanation: "(A one-sentence summary of less than 20 words on how you calculated the VPS for the content.)", + vpm: "(the calculated VPS score.)", + vpm-explanation: "(A one-sentence summary of less than 20 words on how you calculated the VPM for the content.)", } From 3140ca0baccc5615ae65f4777857b8045ec6709b Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 21:21:20 -0800 Subject: [PATCH 22/25] Added /helpers/vm which downloads youtube transcripts and accurate durations of videos using your own YouTube API key. --- helpers/vm | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100755 helpers/vm diff --git a/helpers/vm b/helpers/vm new file mode 100755 index 0000000..815cf0d --- /dev/null +++ b/helpers/vm @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 + +import sys +import re +from googleapiclient.discovery import build +from googleapiclient.errors import HttpError +from youtube_transcript_api import YouTubeTranscriptApi +from dotenv import load_dotenv +import os +import json +import isodate +import argparse + +def get_video_id(url): + # Extract video ID from URL + pattern = r'(?:https?:\/\/)?(?:www\.)?(?:youtube\.com\/(?:[^\/\n\s]+\/\S+\/|(?:v|e(?:mbed)?)\/|\S*?[?&]v=)|youtu\.be\/)([a-zA-Z0-9_-]{11})' + match = re.search(pattern, url) + return match.group(1) if match else None + +def main(url, options): + # Load environment variables from .env file + load_dotenv(os.path.expanduser('~/.config/fabric/.env')) + + # Get YouTube API key from environment variable + api_key = os.getenv('YOUTUBE_API_KEY') + if not api_key: + print("Error: YOUTUBE_API_KEY not found in ~/.config/fabric/.env") + return + + # Extract video ID from URL + video_id = get_video_id(url) + if not video_id: + print("Invalid YouTube URL") + return + + try: + # Initialize the YouTube API client + youtube = build('youtube', 'v3', developerKey=api_key) + + # Get video details + video_response = youtube.videos().list( + id=video_id, + part='contentDetails' + ).execute() + + # Extract video duration and convert to minutes + duration_iso = video_response['items'][0]['contentDetails']['duration'] + duration_seconds = isodate.parse_duration(duration_iso).total_seconds() + duration_minutes = round(duration_seconds / 60) + + # Get video transcript + try: + transcript_list = YouTubeTranscriptApi.get_transcript(video_id) + transcript_text = ' '.join([item['text'] for item in transcript_list]) + transcript_text = transcript_text.replace('\n', ' ') + except Exception as e: + transcript_text = "Transcript not available." + + # Output based on options + if options.duration: + print(duration_minutes) + elif options.transcript: + print(transcript_text) + else: + # Create JSON object + output = { + "transcript": transcript_text, + "duration": duration_minutes + } + # Print JSON object + print(json.dumps(output)) + except HttpError as e: + print("Error: Failed to access YouTube API. Please check your YOUTUBE_API_KEY and ensure it is valid.") + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description='vm (video meta) extracts metadata about a video, such as the transcript and the video\'s duration. By Daniel Miessler') + parser.add_argument('url', nargs='?', help='YouTube video URL') + parser.add_argument('--duration', action='store_true', help='Output only the duration') + parser.add_argument('--transcript', action='store_true', help='Output only the transcript') + args = parser.parse_args() + + if args.url: + main(args.url, args) + else: + parser.print_help() + From e011ecbf138bbad3be58527b8b9c3ade2039f38f Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 21:24:26 -0800 Subject: [PATCH 23/25] Updated vm. --- helpers/vm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/helpers/vm b/helpers/vm index 815cf0d..5e2fa42 100755 --- a/helpers/vm +++ b/helpers/vm @@ -73,7 +73,7 @@ def main(url, options): print("Error: Failed to access YouTube API. Please check your YOUTUBE_API_KEY and ensure it is valid.") if __name__ == '__main__': - parser = argparse.ArgumentParser(description='vm (video meta) extracts metadata about a video, such as the transcript and the video\'s duration. By Daniel Miessler') + parser = argparse.ArgumentParser(description='vm (video meta) extracts metadata about a video, such as the transcript and the video\'s duration. Written by Daniel Miessler.') parser.add_argument('url', nargs='?', help='YouTube video URL') parser.add_argument('--duration', action='store_true', help='Output only the duration') parser.add_argument('--transcript', action='store_true', help='Output only the transcript') From 88332c45b075f89191556c27f19058dbf21b748d Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 21:29:32 -0800 Subject: [PATCH 24/25] Updated to add better docs. --- helpers/vm | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/helpers/vm b/helpers/vm index 5e2fa42..4bd376a 100755 --- a/helpers/vm +++ b/helpers/vm @@ -11,6 +11,21 @@ import json import isodate import argparse +class CustomHelpFormatter(argparse.HelpFormatter): + def _format_usage(self, usage, actions, groups, prefix): + return '' + + def format_help(self): + description = self._root_section.format_help().strip() + usage = self._format_usage( + self._prog, + self._actions, + self._mutually_exclusive_groups, + self._format_usage + ) + usage = usage.replace(self._prog, '').strip() + return f"{description}\n\nUsage: {self._prog} {usage}\n{super().format_help()}" + def get_video_id(url): # Extract video ID from URL pattern = r'(?:https?:\/\/)?(?:www\.)?(?:youtube\.com\/(?:[^\/\n\s]+\/\S+\/|(?:v|e(?:mbed)?)\/|\S*?[?&]v=)|youtu\.be\/)([a-zA-Z0-9_-]{11})' @@ -73,7 +88,10 @@ def main(url, options): print("Error: Failed to access YouTube API. Please check your YOUTUBE_API_KEY and ensure it is valid.") if __name__ == '__main__': - parser = argparse.ArgumentParser(description='vm (video meta) extracts metadata about a video, such as the transcript and the video\'s duration. Written by Daniel Miessler.') + parser = argparse.ArgumentParser( + description='vm (video meta) extracts metadata about a video, such as the transcript and the video\'s duration. By Daniel Miessler', + formatter_class=CustomHelpFormatter + ) parser.add_argument('url', nargs='?', help='YouTube video URL') parser.add_argument('--duration', action='store_true', help='Output only the duration') parser.add_argument('--transcript', action='store_true', help='Output only the transcript') From 58e6e277a6fb34651875372ebfeda35947ab9771 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 14 Feb 2024 21:34:21 -0800 Subject: [PATCH 25/25] Updated vm. --- helpers/vm | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-) diff --git a/helpers/vm b/helpers/vm index 4bd376a..cc19d21 100755 --- a/helpers/vm +++ b/helpers/vm @@ -11,21 +11,6 @@ import json import isodate import argparse -class CustomHelpFormatter(argparse.HelpFormatter): - def _format_usage(self, usage, actions, groups, prefix): - return '' - - def format_help(self): - description = self._root_section.format_help().strip() - usage = self._format_usage( - self._prog, - self._actions, - self._mutually_exclusive_groups, - self._format_usage - ) - usage = usage.replace(self._prog, '').strip() - return f"{description}\n\nUsage: {self._prog} {usage}\n{super().format_help()}" - def get_video_id(url): # Extract video ID from URL pattern = r'(?:https?:\/\/)?(?:www\.)?(?:youtube\.com\/(?:[^\/\n\s]+\/\S+\/|(?:v|e(?:mbed)?)\/|\S*?[?&]v=)|youtu\.be\/)([a-zA-Z0-9_-]{11})' @@ -88,10 +73,7 @@ def main(url, options): print("Error: Failed to access YouTube API. Please check your YOUTUBE_API_KEY and ensure it is valid.") if __name__ == '__main__': - parser = argparse.ArgumentParser( - description='vm (video meta) extracts metadata about a video, such as the transcript and the video\'s duration. By Daniel Miessler', - formatter_class=CustomHelpFormatter - ) + parser = argparse.ArgumentParser(description='vm (video meta) extracts metadata about a video, such as the transcript and the video\'s duration. By Daniel Miessler.') parser.add_argument('url', nargs='?', help='YouTube video URL') parser.add_argument('--duration', action='store_true', help='Output only the duration') parser.add_argument('--transcript', action='store_true', help='Output only the transcript')