From af35483f2d41235ead484e3949a2a7099efe6153 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sat, 7 Jan 2017 16:45:14 +0100 Subject: [PATCH 01/46] lein new duct --- .gitignore | 16 +- README.md | 202 ++++++++++-------- dev/resources/dev.edn | 9 + dev/src/dev.clj | 21 ++ dev/src/user.clj | 8 + project.clj | 40 ++++ .../asciinema/endpoint/example/example.html | 11 + resources/asciinema/errors/404.html | 12 ++ resources/asciinema/errors/500.html | 12 ++ resources/asciinema/public/css/site.css | 103 +++++++++ resources/asciinema/public/favicon.ico | Bin 0 -> 1150 bytes resources/asciinema/public/index.html | 36 ++++ resources/asciinema/public/robots.txt | 2 + resources/asciinema/system.edn | 50 +++++ src/asciinema/endpoint/example.clj | 8 + src/asciinema/main.clj | 15 ++ test/asciinema/endpoint/example_test.clj | 16 ++ 17 files changed, 467 insertions(+), 94 deletions(-) create mode 100644 dev/resources/dev.edn create mode 100644 dev/src/dev.clj create mode 100644 dev/src/user.clj create mode 100644 project.clj create mode 100644 resources/asciinema/endpoint/example/example.html create mode 100644 resources/asciinema/errors/404.html create mode 100644 resources/asciinema/errors/500.html create mode 100644 resources/asciinema/public/css/site.css create mode 100644 resources/asciinema/public/favicon.ico create mode 100644 resources/asciinema/public/index.html create mode 100644 resources/asciinema/public/robots.txt create mode 100644 resources/asciinema/system.edn create mode 100644 src/asciinema/endpoint/example.clj create mode 100644 src/asciinema/main.clj create mode 100644 test/asciinema/endpoint/example_test.clj diff --git a/.gitignore b/.gitignore index bd75f50..12b6dc6 100644 --- a/.gitignore +++ b/.gitignore @@ -23,10 +23,16 @@ public/uploads/* .rbx .env -/.dir-locals.el -/.lein-env -/.lein-repl-history +/target +/classes +/checkouts +pom.xml +pom.xml.asc +*.jar +*.class +/.lein-* /.nrepl-port -/dev/ +/.dir-locals.el /profiles.clj -/target/ +/dev/resources/local.edn +/dev/src/local.clj diff --git a/README.md b/README.md index 548b7fa..4c97863 100644 --- a/README.md +++ b/README.md @@ -1,116 +1,140 @@ -# asciinema.org +# asciinema -[![Build Status](https://travis-ci.org/asciinema/asciinema.org.svg?branch=master)](https://travis-ci.org/asciinema/asciinema.org) -[![Code Climate](https://codeclimate.com/github/asciinema/asciinema.org/badges/gpa.svg)](https://codeclimate.com/github/asciinema/asciinema.org) -[![Coverage Status](https://coveralls.io/repos/asciinema/asciinema.org/badge.svg)](https://coveralls.io/r/asciinema/asciinema.org) +FIXME: description -Record and share your terminal sessions, the right way. +## Developing -asciinema is a free and open source solution for recording terminal sessions -and sharing them on the web. +### Setup -This is the source code of asciinema.org website. You can find asciinema's -terminal recorder at -[asciinema/asciinema](https://github.com/asciinema/asciinema) and asciinema -player at -[asciinema/asciinema-player](https://github.com/asciinema/asciinema-player). - -## Setup instructions - -Below you'll find setup instructions in case you want to contribute, play with -it on your local machine, or setup your own instance for private use or for -your organization. - -### Quickstart Using Docker Compose - -Required: - - [Docker](https://docs.docker.com/engine/getstarted/step_one/#step-1-get-docker) - - [docker-compose 1.5+](https://docs.docker.com/compose/install/) -```bash -$ wget https://raw.githubusercontent.com/asciinema/asciinema.org/master/docker-compose.yml -$ docker-compose up -d asciinema -$ docker-compose run --rm db_init +When you first clone this repository, run: +```sh +lein setup ``` -You can override the address/port that is sent in email with login token by passing HOST="host:port" environment variable when starting the web server. +This will create files for local configuration, and prep your system +for the project. -Assuming you are running Docker Toolbox and VirtualBox: go to http://192.168.99.100:3000/ and enjoy. +### Environment -### Manual setup +To begin developing, start with a REPL. -#### 1. Install dependencies - -asciinema.org site is a Ruby on Rails application. You need to have following -dependencies installed: - -* Ruby 2.0+ (Ruby 2.1 is recommended) - -* bundler gem - `gem install bundler` - -* PostgreSQL 8+ with libpq development headers - `sudo apt-get install postgresql libpq-dev` on Debian/Ubuntu - -* asciinema's libtsm fork (`asciinema` branch) - See [here](https://github.com/asciinema/libtsm/blob/asciinema/README) for installation instructions. - If you don't install it now the setup script (point 4 below) will try to - install it for you anyway. - -* phantomjs 2.0+ - `sudo add-apt-repository ppa:tanguy-patte/phantomjs && sudo apt-get update && sudo apt-get install phantomjs` - -#### 2. Get the source code - -Clone git repository: - -```bash -$ git clone git://github.com/asciinema/asciinema.org.git -$ cd asciinema.org +```sh +lein repl ``` -#### 3. Prepare database config file +Then load the development environment. -Copy *config/database.yml.example* to *config/database.yml*. Then set -database/user/password to whatever you prefer. - -If database specified in database.yml doesn't exist then the following setup -task will create it (make sure database user is allowed to create new -databases). - -#### 4. Setup the app - -Following script will install gem dependencies and setup database: - -```bash -$ ./script/setup +```clojure +user=> (dev) +:loaded ``` -#### 5. Run Rails server +Run `go` to initiate and start the system. -```bash -$ bundle exec rails server +```clojure +dev=> (go) +:started ``` -#### 6. Run the background job processor +By default this creates a web server at . -The background job processor is needed for asciicast pre-processing and -thumbnail generation. +When you make changes to your source files, use `reset` to reload any +modified files and reset the server. -```bash -$ bundle exec sidekiq +```clojure +dev=> (reset) +:reloading (...) +:resumed ``` -## Contributing +### Testing -If you want to contribute to this project check out -[Contributing](http://asciinema.org/contributing) page. +Testing is fastest through the REPL, as you avoid environment startup +time. -## Authors +```clojure +dev=> (test) +... +``` -Developed with passion by [Marcin Kulik](http://ku1ik.com) and great open -source [contributors](https://github.com/asciinema/asciinema.org/contributors) +But you can also run tests through Leiningen. -## Copyright +```sh +lein test +``` -Copyright © 2011-2016 Marcin Kulik. See LICENSE.txt for details. +### Migrations + +Migrations are handled by [ragtime][]. Migration files are stored in +the `resources/migrations` directory, and are applied in alphanumeric +order. + +To update the database to the latest migration, open the REPL and run: + +```clojure +dev=> (migrate) +Applying 20150815144312-create-users +Applying 20150815145033-create-posts +``` + +To rollback the last migration, run: + +```clojure +dev=> (rollback) +Rolling back 20150815145033-create-posts +``` + +Note that the system needs to be setup with `(init)` or `(go)` before +migrations can be applied. + +[ragtime]: https://github.com/weavejester/ragtime + +### Generators + +This project has several generator functions to help you create files. + +To create a new endpoint: + +```clojure +dev=> (gen/endpoint "bar") +Creating file src/foo/endpoint/bar.clj +Creating file test/foo/endpoint/bar_test.clj +Creating directory resources/foo/endpoint/bar +nil +``` + +To create a new component: + +```clojure +dev=> (gen/component "baz") +Creating file src/foo/component/baz.clj +Creating file test/foo/component/baz_test.clj +nil +``` + +To create a new boundary: + +```clojure +dev=> (gen/boundary "quz" foo.component.baz.Baz) +Creating file src/foo/boundary/quz.clj +Creating file test/foo/boundary/quz_test.clj +nil +``` + +To create a new SQL migration: + +```clojure +dev=> (gen/sql-migration "create-users") +Creating file resources/foo/migrations/20160519143643-create-users.up.sql +Creating file resources/foo/migrations/20160519143643-create-users.down.sql +nil +``` + +## Deploying + +FIXME: steps to deploy + +## Legal + +Copyright © 2017 FIXME diff --git a/dev/resources/dev.edn b/dev/resources/dev.edn new file mode 100644 index 0000000..83a8e7c --- /dev/null +++ b/dev/resources/dev.edn @@ -0,0 +1,9 @@ +{:config + {:app + {:middleware + {:functions {:stacktrace #var ring.middleware.stacktrace/wrap-stacktrace} + :applied ^:replace [:not-found :webjars :ring-defaults :route-aliases :stacktrace]}} + :http + {:port 4000} + :db + {:uri "jdbc:postgresql://localhost/postgres"}}} diff --git a/dev/src/dev.clj b/dev/src/dev.clj new file mode 100644 index 0000000..5889c96 --- /dev/null +++ b/dev/src/dev.clj @@ -0,0 +1,21 @@ +(ns dev + (:refer-clojure :exclude [test]) + (:require [clojure.repl :refer :all] + [clojure.pprint :refer [pprint]] + [clojure.tools.namespace.repl :refer [refresh]] + [clojure.java.io :as io] + [com.stuartsierra.component :as component] + [duct.generate :as gen] + [duct.util.repl :refer [setup test cljs-repl migrate rollback]] + [duct.util.system :refer [load-system]] + [reloaded.repl :refer [system init start stop go reset]])) + +(defn new-system [] + (load-system (keep io/resource ["asciinema/system.edn" "dev.edn" "local.edn"]))) + +(when (io/resource "local.clj") + (load "local")) + +(gen/set-ns-prefix 'asciinema) + +(reloaded.repl/set-init! new-system) diff --git a/dev/src/user.clj b/dev/src/user.clj new file mode 100644 index 0000000..9cf3c0c --- /dev/null +++ b/dev/src/user.clj @@ -0,0 +1,8 @@ +(ns user) + +(defn dev + "Load and switch to the 'dev' namespace." + [] + (require 'dev) + (in-ns 'dev) + :loaded) diff --git a/project.clj b/project.clj new file mode 100644 index 0000000..d886b2c --- /dev/null +++ b/project.clj @@ -0,0 +1,40 @@ +(defproject asciinema "0.1.0-SNAPSHOT" + :description "FIXME: write description" + :url "http://example.com/FIXME" + :min-lein-version "2.0.0" + :dependencies [[org.clojure/clojure "1.8.0"] + [com.stuartsierra/component "0.3.1"] + [compojure "1.5.1"] + [duct "0.8.2"] + [environ "1.1.0"] + [ring "1.5.0"] + [ring/ring-defaults "0.2.1"] + [ring-jetty-component "0.3.1"] + [ring-webjars "0.1.1"] + [org.slf4j/slf4j-nop "1.7.21"] + [org.webjars/normalize.css "3.0.2"] + [duct/hikaricp-component "0.1.0"] + [org.postgresql/postgresql "9.4.1211"] + [duct/ragtime-component "0.1.4"]] + :plugins [[lein-environ "1.0.3"]] + :main ^:skip-aot asciinema.main + :target-path "target/%s/" + :aliases {"setup" ["run" "-m" "duct.util.repl/setup"]} + :profiles + {:dev [:project/dev :profiles/dev] + :test [:project/test :profiles/test] + :uberjar {:aot :all} + :profiles/dev {} + :profiles/test {} + :project/dev {:dependencies [[duct/generate "0.8.2"] + [reloaded.repl "0.2.3"] + [org.clojure/tools.namespace "0.2.11"] + [org.clojure/tools.nrepl "0.2.12"] + [eftest "0.1.1"] + [com.gearswithingears/shrubbery "0.4.1"] + [kerodon "0.8.0"]] + :source-paths ["dev/src"] + :resource-paths ["dev/resources"] + :repl-options {:init-ns user} + :env {:port "3000"}} + :project/test {}}) diff --git a/resources/asciinema/endpoint/example/example.html b/resources/asciinema/endpoint/example/example.html new file mode 100644 index 0000000..f808576 --- /dev/null +++ b/resources/asciinema/endpoint/example/example.html @@ -0,0 +1,11 @@ + + + + Example Endpoint + + + + +

This is an example endpoint

+ + diff --git a/resources/asciinema/errors/404.html b/resources/asciinema/errors/404.html new file mode 100644 index 0000000..b9cab04 --- /dev/null +++ b/resources/asciinema/errors/404.html @@ -0,0 +1,12 @@ + + + + Server Error + + + + +

Resource Not Found

+

The requested page does not exist.

+ + diff --git a/resources/asciinema/errors/500.html b/resources/asciinema/errors/500.html new file mode 100644 index 0000000..985ff84 --- /dev/null +++ b/resources/asciinema/errors/500.html @@ -0,0 +1,12 @@ + + + + Server Error + + + + +

Internal Server Error

+

Sorry, something went wrong.

+ + diff --git a/resources/asciinema/public/css/site.css b/resources/asciinema/public/css/site.css new file mode 100644 index 0000000..37086ca --- /dev/null +++ b/resources/asciinema/public/css/site.css @@ -0,0 +1,103 @@ +.error-page body { + background: #eee; +} + +.error-page h1 { + margin: 15% 0 0 0; + text-align: center; + font-size: 42px; + color: #900; +} + +.error-page h2 { + text-align: center; + font-size: 32px; + font-weight: normal; + color: #333; +} + +.welcome body { + background: #eee; + color: #333; + font-family: Helvetica, Arial, sans-serif; + max-width: 700px; + padding: 15px; + margin: auto; +} + +.welcome p { + line-height: 1.4em; +} + +.welcome code { + font-family: Menlo, DejaVu Sans Mono, Lucida Console, monospace; + font-size: 12px; + background: #ddd; + color: #111; +} + +.welcome h1 { + text-align: center; + font-size: 36px; + font-weight: lighter; + margin: 40px 0 30px 0; +} + +.welcome h1 .outer { + border: solid 4px #555; + padding: 3px; + display: inline-block; +} + +.welcome h1 .inner { + border: solid 2px #555; + padding: 0 3px; + display: inline-block; + font-weight: normal; + color: #444; +} + +.welcome .project-name { + font-weight: bold; +} + +.welcome .profiles { + margin-top: 30px; +} + +.welcome .profiles code { + font-size: 11px; +} + +.welcome .profiles h2 { + font-weight: normal; + font-size: 23px; + margin-bottom: 0; + color: #333; +} + +.welcome .profiles dl { + margin: 0 10px; +} + +.welcome .profiles dt { + font-weight: normal; + font-size: 19px; + margin: 18px 0 5px 0; +} + +.welcome .profiles dd { + font-size: 14px; + margin: 8px 0 8px 0; +} + +.example body { + background: #eee; +} + +.example h1 { + margin: 15% 0 0 0; + text-align: center; + font-size: 36px; + font-weight: normal; +} diff --git a/resources/asciinema/public/favicon.ico b/resources/asciinema/public/favicon.ico new file mode 100644 index 0000000000000000000000000000000000000000..0e50cb2fb96b2ae63a2cf81adf4e1b3869cfe152 GIT binary patch literal 1150 zcmeH_I}XAy5JV>uO}bRnlpHO`paEJsO3ud-sKSn{C#|d)gasXrHshU-6Rj)_@l2EA zz0A@v# + + + Welcome to Duct + + + + +

Welcome to Duct

+
+

Congratulations! Your project asciinema is + ready and running.

+

This is a static welcome page located at resources/asciinema/public/index.html + in the project directory. Remove or replace it when you start developing. + If you remove the index page entirely, be sure to change the + :route-aliases map in resources/asciinema/system.edn. +

+
+

Template profiles used:

+
+
+example
+
Adds an example endpoint at /example.
+
+postgres
+
Adds a PostgreSQL dependency and database component. The database used for + development defaults to postgres on localhost.
+
+ragtime
+
Adds Ragtime migrations. Use (migrate) and (rollback) + in the REPL. Migrations are stored in resources/asciinema/migrations. +
+
+site
+
Adds middleware and configuration suited for a user-facing website.
+
+
+ + + diff --git a/resources/asciinema/public/robots.txt b/resources/asciinema/public/robots.txt new file mode 100644 index 0000000..eb05362 --- /dev/null +++ b/resources/asciinema/public/robots.txt @@ -0,0 +1,2 @@ +User-agent: * +Disallow: diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn new file mode 100644 index 0000000..a61ca89 --- /dev/null +++ b/resources/asciinema/system.edn @@ -0,0 +1,50 @@ +{:components + {:app #var duct.component.handler/handler-component + :http #var ring.component.jetty/jetty-server + :db #var duct.component.hikaricp/hikaricp + :ragtime #var duct.component.ragtime/ragtime} + :endpoints + {:example #var asciinema.endpoint.example/example-endpoint} + :dependencies + {:http [:app] + :app [:example] + :ragtime [:db] + :example [:db]} + :config + {:app + {:middleware + {:functions + {:hide-errors #var duct.middleware.errors/wrap-hide-errors + :not-found #var duct.middleware.not-found/wrap-not-found + :ring-defaults #var ring.middleware.defaults/wrap-defaults + :route-aliases #var duct.middleware.route-aliases/wrap-route-aliases + :webjars #var ring.middleware.webjars/wrap-webjars} + :applied + [:not-found :webjars :ring-defaults :route-aliases :hide-errors] + :arguments + {:not-found #resource "asciinema/errors/404.html" + :hide-errors #resource "asciinema/errors/500.html" + :route-aliases {"/" "/index.html"} + :ring-defaults + {:params {:urlencoded true + :keywordize true + :multipart true + :nested true} + :cookies true + :session {:flash true + :cookie-attrs {:http-only true}} + :security {:anti-forgery true + :xss-protection {:enable? true, :mode :block} + :frame-options :sameorigin + :content-type-options :nosniff} + :static {:resources "asciinema/public"} + :responses {:not-modified-responses true + :absolute-redirects true + :content-types true + :default-charset "utf-8"}}}}} + :http + {:port http-port} + :db + {:uri db-uri} + :ragtime + {:resource-path "asciinema/migrations"}}} diff --git a/src/asciinema/endpoint/example.clj b/src/asciinema/endpoint/example.clj new file mode 100644 index 0000000..46c9821 --- /dev/null +++ b/src/asciinema/endpoint/example.clj @@ -0,0 +1,8 @@ +(ns asciinema.endpoint.example + (:require [compojure.core :refer :all] + [clojure.java.io :as io])) + +(defn example-endpoint [{{db :spec} :db}] + (context "/example" [] + (GET "/" [] + (io/resource "asciinema/endpoint/example/example.html")))) diff --git a/src/asciinema/main.clj b/src/asciinema/main.clj new file mode 100644 index 0000000..becb032 --- /dev/null +++ b/src/asciinema/main.clj @@ -0,0 +1,15 @@ +(ns asciinema.main + (:gen-class) + (:require [com.stuartsierra.component :as component] + [duct.util.runtime :refer [add-shutdown-hook]] + [duct.util.system :refer [load-system]] + [environ.core :refer [env]] + [clojure.java.io :as io])) + +(defn -main [& args] + (let [bindings {'http-port (Integer/parseInt (:port env "3000")) + 'db-uri (:database-url env)} + system (->> (load-system [(io/resource "asciinema/system.edn")] bindings) + (component/start))] + (add-shutdown-hook ::stop-system #(component/stop system)) + (println "Started HTTP server on port" (-> system :http :port)))) diff --git a/test/asciinema/endpoint/example_test.clj b/test/asciinema/endpoint/example_test.clj new file mode 100644 index 0000000..ec32ab1 --- /dev/null +++ b/test/asciinema/endpoint/example_test.clj @@ -0,0 +1,16 @@ +(ns asciinema.endpoint.example-test + (:require [com.stuartsierra.component :as component] + [clojure.test :refer :all] + [kerodon.core :refer :all] + [kerodon.test :refer :all] + [shrubbery.core :as shrub] + [asciinema.endpoint.example :as example])) + +(def handler + (example/example-endpoint {})) + +(deftest smoke-test + (testing "example page exists" + (-> (session handler) + (visit "/example") + (has (status? 200) "page exists")))) From ffefc44f41cf0a90dbe0053d4a1e7213720edb15 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Mon, 9 Jan 2017 19:33:01 +0100 Subject: [PATCH 02/46] Switch to aleph --- project.clj | 1 + resources/asciinema/system.edn | 2 +- src/asciinema/component/aleph.clj | 17 +++++++++++++++++ 3 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 src/asciinema/component/aleph.clj diff --git a/project.clj b/project.clj index d886b2c..c35c7cd 100644 --- a/project.clj +++ b/project.clj @@ -11,6 +11,7 @@ [ring/ring-defaults "0.2.1"] [ring-jetty-component "0.3.1"] [ring-webjars "0.1.1"] + [aleph "0.4.1"] [org.slf4j/slf4j-nop "1.7.21"] [org.webjars/normalize.css "3.0.2"] [duct/hikaricp-component "0.1.0"] diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index a61ca89..6d3185e 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -1,6 +1,6 @@ {:components {:app #var duct.component.handler/handler-component - :http #var ring.component.jetty/jetty-server + :http #var asciinema.component.aleph/aleph-server :db #var duct.component.hikaricp/hikaricp :ragtime #var duct.component.ragtime/ragtime} :endpoints diff --git a/src/asciinema/component/aleph.clj b/src/asciinema/component/aleph.clj new file mode 100644 index 0000000..a77bfbc --- /dev/null +++ b/src/asciinema/component/aleph.clj @@ -0,0 +1,17 @@ +(ns asciinema.component.aleph + (:require [com.stuartsierra.component :as component] + [aleph.http :refer [start-server]])) + +(defrecord WebServer [port server app] + component/Lifecycle + (start [component] + (let [handler (:handler app) + server (start-server handler {:port port :join? false})] + (assoc component :server server))) + (stop [component] + (when server + (.close server) + component))) + +(defn aleph-server [{:keys [port app]}] + (map->WebServer {:port port :app app})) From 08f2c2fdae8a217301d5dedad5f70b54119924a3 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Fri, 13 Jan 2017 19:10:34 +0100 Subject: [PATCH 03/46] Block main thread after starting system --- src/asciinema/main.clj | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/asciinema/main.clj b/src/asciinema/main.clj index becb032..459fc60 100644 --- a/src/asciinema/main.clj +++ b/src/asciinema/main.clj @@ -12,4 +12,5 @@ system (->> (load-system [(io/resource "asciinema/system.edn")] bindings) (component/start))] (add-shutdown-hook ::stop-system #(component/stop system)) - (println "Started HTTP server on port" (-> system :http :port)))) + (println "Started HTTP server on port" (-> system :http :port))) + @(promise)) From 8b2f76440723b67d0db96179b8f587dae94ccf20 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sun, 15 Jan 2017 19:11:22 +0100 Subject: [PATCH 04/46] FileStore and FileServer boundaries --- dev/resources/dev.edn | 11 ++++++-- dev/src/dev.clj | 6 ++++- project.clj | 2 ++ resources/asciinema/system.edn | 16 ++++++++++-- src/asciinema/boundary/file_server.clj | 4 +++ src/asciinema/boundary/file_store.clj | 7 ++++++ src/asciinema/component/local_file_server.clj | 12 +++++++++ src/asciinema/component/local_file_store.clj | 25 +++++++++++++++++++ src/asciinema/component/s3_file_server.clj | 15 +++++++++++ src/asciinema/component/s3_file_store.clj | 25 +++++++++++++++++++ test/asciinema/boundary/file_server_test.clj | 7 ++++++ test/asciinema/boundary/file_store_test.clj | 7 ++++++ 12 files changed, 132 insertions(+), 5 deletions(-) create mode 100644 src/asciinema/boundary/file_server.clj create mode 100644 src/asciinema/boundary/file_store.clj create mode 100644 src/asciinema/component/local_file_server.clj create mode 100644 src/asciinema/component/local_file_store.clj create mode 100644 src/asciinema/component/s3_file_server.clj create mode 100644 src/asciinema/component/s3_file_store.clj create mode 100644 test/asciinema/boundary/file_server_test.clj create mode 100644 test/asciinema/boundary/file_store_test.clj diff --git a/dev/resources/dev.edn b/dev/resources/dev.edn index 83a8e7c..3ee655c 100644 --- a/dev/resources/dev.edn +++ b/dev/resources/dev.edn @@ -1,4 +1,9 @@ -{:config +{:components + {:file-store #var asciinema.component.local-file-store/local-file-store + :file-server #var asciinema.component.local-file-server/local-file-server} + :dependencies + {:file-server [:file-store]} + :config {:app {:middleware {:functions {:stacktrace #var ring.middleware.stacktrace/wrap-stacktrace} @@ -6,4 +11,6 @@ :http {:port 4000} :db - {:uri "jdbc:postgresql://localhost/postgres"}}} + {:uri "jdbc:postgresql://localhost:15432/asciinema_development?user=vagrant"} + :file-store + {:path "uploads/"}}} diff --git a/dev/src/dev.clj b/dev/src/dev.clj index 5889c96..b720e61 100644 --- a/dev/src/dev.clj +++ b/dev/src/dev.clj @@ -8,7 +8,11 @@ [duct.generate :as gen] [duct.util.repl :refer [setup test cljs-repl migrate rollback]] [duct.util.system :refer [load-system]] - [reloaded.repl :refer [system init start stop go reset]])) + [reloaded.repl :refer [system init start stop go reset]] + [asciinema.boundary.file-server :as file-server] + [asciinema.boundary.file-store :as file-store] + [asciinema.component.local-file-store :refer [->LocalFileStore]] + [asciinema.component.local-file-server :refer [->LocalFileServer]])) (defn new-system [] (load-system (keep io/resource ["asciinema/system.edn" "dev.edn" "local.edn"]))) diff --git a/project.clj b/project.clj index c35c7cd..1fd9213 100644 --- a/project.clj +++ b/project.clj @@ -11,6 +11,8 @@ [ring/ring-defaults "0.2.1"] [ring-jetty-component "0.3.1"] [ring-webjars "0.1.1"] + [metosin/ring-http-response "0.8.1"] + [clj-aws-s3 "0.3.10" :exclusions [clj-time]] [aleph "0.4.1"] [org.slf4j/slf4j-nop "1.7.21"] [org.webjars/normalize.css "3.0.2"] diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index 6d3185e..84aa5df 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -2,7 +2,9 @@ {:app #var duct.component.handler/handler-component :http #var asciinema.component.aleph/aleph-server :db #var duct.component.hikaricp/hikaricp - :ragtime #var duct.component.ragtime/ragtime} + :ragtime #var duct.component.ragtime/ragtime + :file-store #var asciinema.component.s3-file-store/s3-file-store + :file-server #var asciinema.component.s3-file-server/s3-file-server} :endpoints {:example #var asciinema.endpoint.example/example-endpoint} :dependencies @@ -47,4 +49,14 @@ :db {:uri db-uri} :ragtime - {:resource-path "asciinema/migrations"}}} + {:resource-path "asciinema/migrations"} + :file-store + {:cred {:access-key s3-access-key + :secret-key s3-secret-key} + :bucket s3-bucket + :path-prefix "uploads/"} + :file-server + {:cred {:access-key s3-access-key + :secret-key s3-secret-key} + :bucket s3-bucket + :path-prefix "uploads/"}}} diff --git a/src/asciinema/boundary/file_server.clj b/src/asciinema/boundary/file_server.clj new file mode 100644 index 0000000..8fb06bd --- /dev/null +++ b/src/asciinema/boundary/file_server.clj @@ -0,0 +1,4 @@ +(ns asciinema.boundary.file-server) + +(defprotocol FileServer + (serve [this path])) diff --git a/src/asciinema/boundary/file_store.clj b/src/asciinema/boundary/file_store.clj new file mode 100644 index 0000000..d9107a8 --- /dev/null +++ b/src/asciinema/boundary/file_store.clj @@ -0,0 +1,7 @@ +(ns asciinema.boundary.file-store) + +(defprotocol FileStore + (put-file [this file path] [this file path size]) + (input-stream [this path]) + (move-file [this old-path new-path]) + (delete-file [this path])) diff --git a/src/asciinema/component/local_file_server.clj b/src/asciinema/component/local_file_server.clj new file mode 100644 index 0000000..4dbeaa9 --- /dev/null +++ b/src/asciinema/component/local_file_server.clj @@ -0,0 +1,12 @@ +(ns asciinema.component.local-file-server + (:require [asciinema.boundary.file-server :as file-server] + [asciinema.boundary.file-store :as file-store] + [ring.util.http-response :as response])) + +(defrecord LocalFileServer [file-store] + file-server/FileServer + (serve [this path] + (response/ok (file-store/input-stream file-store path)))) + +(defn local-file-server [{:keys [file-store]}] + (->LocalFileServer file-store)) diff --git a/src/asciinema/component/local_file_store.clj b/src/asciinema/component/local_file_store.clj new file mode 100644 index 0000000..e250f1d --- /dev/null +++ b/src/asciinema/component/local_file_store.clj @@ -0,0 +1,25 @@ +(ns asciinema.component.local-file-store + (:require [asciinema.boundary.file-store :as file-store] + [clojure.java.io :as io])) + +(defrecord LocalFileStore [base-path] + file-store/FileStore + (put-file [this file path] + (let [path (str base-path path)] + (io/make-parents path) + (io/copy file (io/file path)))) + (put-file [this file path size] + (file-store/put-file this file path)) + (input-stream [this path] + (let [path (str base-path path)] + (io/input-stream path))) + (move-file [this old-path new-path] + (let [old-path (str base-path old-path) + new-path (str base-path new-path)] + (.renameTo (io/file old-path) (io/file new-path)))) + (delete-file [this path] + (let [path (str base-path path)] + (io/delete-file path)))) + +(defn local-file-store [{:keys [path]}] + (->LocalFileStore path)) diff --git a/src/asciinema/component/s3_file_server.clj b/src/asciinema/component/s3_file_server.clj new file mode 100644 index 0000000..c4163a7 --- /dev/null +++ b/src/asciinema/component/s3_file_server.clj @@ -0,0 +1,15 @@ +(ns asciinema.component.s3-file-server + (:require [asciinema.boundary.file-server :as file-server] + [aws.sdk.s3 :as s3] + [ring.util.http-response :as response])) + +;; TODO support custom expiry date (it's 1 day now) + +(defrecord S3FileServer [cred bucket path-prefix] + file-server/FileServer + (serve [this path] + (let [path (str path-prefix path)] + (response/found (s3/generate-presigned-url cred bucket path))))) + +(defn s3-file-server [{:keys [cred bucket path-prefix]}] + (->S3FileServer cred bucket path-prefix)) diff --git a/src/asciinema/component/s3_file_store.clj b/src/asciinema/component/s3_file_store.clj new file mode 100644 index 0000000..e9fb608 --- /dev/null +++ b/src/asciinema/component/s3_file_store.clj @@ -0,0 +1,25 @@ +(ns asciinema.component.s3-file-store + (:require [asciinema.boundary.file-store :as file-store] + [aws.sdk.s3 :as s3])) + +(defrecord S3FileStore [cred bucket path-prefix] + file-store/FileStore + (put-file [this file path] + (file-store/put-file this file path nil)) + (put-file [this file path size] + (let [path (str path-prefix path)] + (s3/put-object cred bucket path file {:content-length size}))) + (input-stream [this path] + (let [path (str path-prefix path)] + (:content (s3/get-object cred bucket path)))) + (move-file [this old-path new-path] + (let [old-path (str path-prefix old-path) + new-path (str path-prefix new-path)] + (s3/copy-object cred bucket old-path new-path) + (s3/delete-object cred bucket old-path))) + (delete-file [this path] + (let [path (str path-prefix path)] + (s3/delete-object cred bucket path)))) + +(defn s3-file-store [{:keys [cred bucket path-prefix]}] + (->S3FileStore cred bucket path-prefix)) diff --git a/test/asciinema/boundary/file_server_test.clj b/test/asciinema/boundary/file_server_test.clj new file mode 100644 index 0000000..2136484 --- /dev/null +++ b/test/asciinema/boundary/file_server_test.clj @@ -0,0 +1,7 @@ +(ns asciinema.boundary.file-server-test + (:require [clojure.test :refer :all] + [asciinema.boundary.file-server :as file-server])) + +(deftest a-test + (testing "FIXME, I fail." + (is (= 0 1)))) diff --git a/test/asciinema/boundary/file_store_test.clj b/test/asciinema/boundary/file_store_test.clj new file mode 100644 index 0000000..b6d3786 --- /dev/null +++ b/test/asciinema/boundary/file_store_test.clj @@ -0,0 +1,7 @@ +(ns asciinema.boundary.file-store-test + (:require [clojure.test :refer :all] + [asciinema.boundary.file-store :as file-store])) + +(deftest a-test + (testing "FIXME, I fail." + (is (= 0 1)))) From dca6f92e6b7ab46ab0def30604f507487d3643fb Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sun, 15 Jan 2017 19:30:23 +0100 Subject: [PATCH 05/46] Fix clj-aws-s3 dependencies --- project.clj | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/project.clj b/project.clj index 1fd9213..3cec63c 100644 --- a/project.clj +++ b/project.clj @@ -4,15 +4,16 @@ :min-lein-version "2.0.0" :dependencies [[org.clojure/clojure "1.8.0"] [com.stuartsierra/component "0.3.1"] - [compojure "1.5.1"] + [metosin/ring-http-response "0.8.1"] + [clj-time "0.12.0"] [duct "0.8.2"] + [compojure "1.5.2"] [environ "1.1.0"] [ring "1.5.0"] [ring/ring-defaults "0.2.1"] [ring-jetty-component "0.3.1"] [ring-webjars "0.1.1"] - [metosin/ring-http-response "0.8.1"] - [clj-aws-s3 "0.3.10" :exclusions [clj-time]] + [clj-aws-s3 "0.3.10" :exclusions [joda-time com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.core/jackson-annotations]] [aleph "0.4.1"] [org.slf4j/slf4j-nop "1.7.21"] [org.webjars/normalize.css "3.0.2"] From 70aedbf12c82ab4da8c358823169e8b28298293c Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sat, 11 Feb 2017 17:50:57 +0100 Subject: [PATCH 06/46] AsciicastDatabase component --- src/asciinema/boundary/asciicast_database.clj | 36 +++++++++++++ .../boundary/asciicast_database_test.clj | 53 +++++++++++++++++++ 2 files changed, 89 insertions(+) create mode 100644 src/asciinema/boundary/asciicast_database.clj create mode 100644 test/asciinema/boundary/asciicast_database_test.clj diff --git a/src/asciinema/boundary/asciicast_database.clj b/src/asciinema/boundary/asciicast_database.clj new file mode 100644 index 0000000..8fb904d --- /dev/null +++ b/src/asciinema/boundary/asciicast_database.clj @@ -0,0 +1,36 @@ +(ns asciinema.boundary.asciicast-database + (:require [clojure.java.jdbc :as jdbc] + [clj-time.coerce :as timec] + [duct.component.hikaricp :as hikaricp])) + +(defprotocol AsciicastDatabase + (get-asciicast-by-id [this id]) + (get-asciicast-by-token [this token])) + +(extend-protocol clojure.java.jdbc/ISQLValue + org.joda.time.DateTime + (sql-value [val] + (timec/to-sql-time val))) + +(extend-protocol clojure.java.jdbc/IResultSetReadColumn + java.sql.Timestamp + (result-set-read-column [x _ _] + (timec/from-sql-time x))) + +(def q-get-asciicast-by-id "SELECT * FROM asciicasts WHERE id=?") +(def q-get-asciicast-by-secret-token "SELECT * FROM asciicasts WHERE secret_token=?") +(def q-get-public-asciicast-by-id "SELECT * FROM asciicasts WHERE id=? AND private=FALSE") + +(extend-protocol AsciicastDatabase + duct.component.hikaricp.HikariCP + + (get-asciicast-by-id [{db :spec} id] + (first (jdbc/query db [q-get-asciicast-by-id id]))) + + (get-asciicast-by-token [{db :spec} token] + (when-let [query (cond + (re-matches #"\d+" token) + [q-get-public-asciicast-by-id (Long/parseLong token)] + (= (count token) 25) + [q-get-asciicast-by-secret-token token])] + (first (jdbc/query db query))))) diff --git a/test/asciinema/boundary/asciicast_database_test.clj b/test/asciinema/boundary/asciicast_database_test.clj new file mode 100644 index 0000000..5860bad --- /dev/null +++ b/test/asciinema/boundary/asciicast_database_test.clj @@ -0,0 +1,53 @@ +(ns asciinema.boundary.asciicast-database-test + (:require [clojure.test :refer :all] + [clojure.java.jdbc :as jdbc] + [clj-time.local :as timel] + [com.stuartsierra.component :as component] + [asciinema.boundary.asciicast-database :as db])) + +(defmacro with-db-component [component-var & body] + `(let [component# (-> (duct.component.hikaricp/hikaricp {:uri "jdbc:postgresql://localhost:15432/asciinema_test?user=vagrant"}) + component/start)] + (try + (jdbc/with-db-transaction [db# (:spec component#)] + (let [~component-var (assoc component# :spec db#)] + (jdbc/db-set-rollback-only! db#) + ~@body)) + (finally + (component/stop component#))))) + +(defn insert-asciicast + ([db] (insert-asciicast db {})) + ([db attrs] + (first (jdbc/insert! db :asciicasts (merge {:duration 10.0 + :terminal_columns 80 + :terminal_lines 24 + :created_at (timel/local-now) + :updated_at (timel/local-now) + :version 1 + :secret_token "abcdeabcdeabcdeabcdeabcde"} + attrs))))) + +(deftest get-asciicast-by-id-test + (testing "for existing asciicast" + (with-db-component db + (let [asciicast (insert-asciicast (:spec db))] + (is (map? (db/get-asciicast-by-id db (:id asciicast))))))) + (testing "for non-existing asciicast" + (with-db-component db + (is (nil? (db/get-asciicast-by-id db 1)))))) + +(deftest get-asciicast-by-token-test + (testing "for existing public asciicast" + (with-db-component db + (let [asciicast (insert-asciicast (:spec db) {:private false})] + (is (map? (db/get-asciicast-by-token db (:secret_token asciicast)))) + (is (map? (db/get-asciicast-by-token db (-> asciicast :id str))))))) + (testing "for existing private asciicast" + (with-db-component db + (let [asciicast (insert-asciicast (:spec db) {:private true})] + (is (map? (db/get-asciicast-by-token db (:secret_token asciicast)))) + (is (nil? (db/get-asciicast-by-token db (-> asciicast :id str))))))) + (testing "for non-existing asciicast" + (with-db-component db + (is (nil? (db/get-asciicast-by-token db "1")))))) From c1c1ffde7027e06b7f14ed636b00ddc897f1a798 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Mon, 13 Feb 2017 19:43:02 +0100 Subject: [PATCH 07/46] Extend FileServer to accept options (:expires, :filename) --- src/asciinema/boundary/file_server.clj | 2 +- src/asciinema/component/local_file_server.clj | 7 ++++- src/asciinema/component/s3_file_server.clj | 29 +++++++++++++++++-- 3 files changed, 33 insertions(+), 5 deletions(-) diff --git a/src/asciinema/boundary/file_server.clj b/src/asciinema/boundary/file_server.clj index 8fb06bd..c380724 100644 --- a/src/asciinema/boundary/file_server.clj +++ b/src/asciinema/boundary/file_server.clj @@ -1,4 +1,4 @@ (ns asciinema.boundary.file-server) (defprotocol FileServer - (serve [this path])) + (serve [this path] [this path opts])) diff --git a/src/asciinema/component/local_file_server.clj b/src/asciinema/component/local_file_server.clj index 4dbeaa9..70b6df7 100644 --- a/src/asciinema/component/local_file_server.clj +++ b/src/asciinema/component/local_file_server.clj @@ -6,7 +6,12 @@ (defrecord LocalFileServer [file-store] file-server/FileServer (serve [this path] - (response/ok (file-store/input-stream file-store path)))) + (file-server/serve this path {})) + (serve [this path {:keys [filename]}] + (let [resp (response/ok (file-store/input-stream file-store path))] + (if filename + (response/header resp "Content-Disposition" (str "attachment; filename=" filename)) + resp)))) (defn local-file-server [{:keys [file-store]}] (->LocalFileServer file-store)) diff --git a/src/asciinema/component/s3_file_server.clj b/src/asciinema/component/s3_file_server.clj index c4163a7..5529922 100644 --- a/src/asciinema/component/s3_file_server.clj +++ b/src/asciinema/component/s3_file_server.clj @@ -1,15 +1,38 @@ (ns asciinema.component.s3-file-server (:require [asciinema.boundary.file-server :as file-server] [aws.sdk.s3 :as s3] - [ring.util.http-response :as response])) + [clj-time.core :as time] + [clj-time.coerce :as timec] + [ring.util.http-response :as response]) + (:import com.amazonaws.services.s3.model.ResponseHeaderOverrides + com.amazonaws.services.s3.AmazonS3Client + com.amazonaws.auth.BasicAWSCredentials + com.amazonaws.services.s3.model.GeneratePresignedUrlRequest)) -;; TODO support custom expiry date (it's 1 day now) +(defn- s3-client* [cred] + (let [credentials (BasicAWSCredentials. (:access-key cred) (:secret-key cred))] + (AmazonS3Client. credentials))) + +(def ^:private s3-client (memoize s3-client*)) + +(defn- generate-presigned-url [cred bucket path {:keys [expires filename] + :or {expires (-> 1 time/days time/from-now)}}] + (let [client (s3-client cred) + request (GeneratePresignedUrlRequest. bucket path)] + (.setExpiration request (timec/to-date expires)) + (when filename + (let [header-overrides (doto (ResponseHeaderOverrides.) + (.setContentDisposition (str "attachment; filename=" filename)))] + (.setResponseHeaders request header-overrides))) + (.toString (.generatePresignedUrl client request)))) (defrecord S3FileServer [cred bucket path-prefix] file-server/FileServer (serve [this path] + (file-server/serve this path {})) + (serve [this path opts] (let [path (str path-prefix path)] - (response/found (s3/generate-presigned-url cred bucket path))))) + (response/found (generate-presigned-url cred bucket path opts))))) (defn s3-file-server [{:keys [cred bucket path-prefix]}] (->S3FileServer cred bucket path-prefix)) From 527db9dd9b60c8aeba0a76288b90d4a800544979 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Mon, 13 Feb 2017 19:49:17 +0100 Subject: [PATCH 08/46] cljr-clean-ns --- src/asciinema/component/local_file_server.clj | 5 +++-- src/asciinema/component/s3_file_server.clj | 11 +++++------ 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/src/asciinema/component/local_file_server.clj b/src/asciinema/component/local_file_server.clj index 70b6df7..eeab31d 100644 --- a/src/asciinema/component/local_file_server.clj +++ b/src/asciinema/component/local_file_server.clj @@ -1,6 +1,7 @@ (ns asciinema.component.local-file-server - (:require [asciinema.boundary.file-server :as file-server] - [asciinema.boundary.file-store :as file-store] + (:require [asciinema.boundary + [file-server :as file-server] + [file-store :as file-store]] [ring.util.http-response :as response])) (defrecord LocalFileServer [file-store] diff --git a/src/asciinema/component/s3_file_server.clj b/src/asciinema/component/s3_file_server.clj index 5529922..2c04a39 100644 --- a/src/asciinema/component/s3_file_server.clj +++ b/src/asciinema/component/s3_file_server.clj @@ -1,13 +1,12 @@ (ns asciinema.component.s3-file-server (:require [asciinema.boundary.file-server :as file-server] - [aws.sdk.s3 :as s3] - [clj-time.core :as time] - [clj-time.coerce :as timec] + [clj-time + [coerce :as timec] + [core :as time]] [ring.util.http-response :as response]) - (:import com.amazonaws.services.s3.model.ResponseHeaderOverrides + (:import com.amazonaws.auth.BasicAWSCredentials com.amazonaws.services.s3.AmazonS3Client - com.amazonaws.auth.BasicAWSCredentials - com.amazonaws.services.s3.model.GeneratePresignedUrlRequest)) + [com.amazonaws.services.s3.model GeneratePresignedUrlRequest ResponseHeaderOverrides])) (defn- s3-client* [cred] (let [credentials (BasicAWSCredentials. (:access-key cred) (:secret-key cred))] From be4995c9f93058b999c7a64876575ef2a6a034b6 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Mon, 13 Feb 2017 19:52:53 +0100 Subject: [PATCH 09/46] Add asciicast model ns --- src/asciinema/model/asciicast.clj | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 src/asciinema/model/asciicast.clj diff --git a/src/asciinema/model/asciicast.clj b/src/asciinema/model/asciicast.clj new file mode 100644 index 0000000..1d0cfed --- /dev/null +++ b/src/asciinema/model/asciicast.clj @@ -0,0 +1,13 @@ +(ns asciinema.model.asciicast) + +(defn json-store-path [{:keys [id file stdout_frames]}] + (cond + file (str "asciicast/file/" id "/" file) + stdout_frames (str "asciicast/stdout_frames/" id "/" stdout_frames))) + +(def default-theme "asciinema") + +(defn theme-name [asciicast user] + (or (:theme_name asciicast) ; ensure we don't store empty strings in db + (:theme_name user) + default-theme)) From 0eda5c040c7450054ac00be34d5b63bf315566b9 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 14 Feb 2017 12:15:51 +0100 Subject: [PATCH 10/46] Endpoint for serving recording JSON file --- project.clj | 4 +++- resources/asciinema/system.edn | 6 +++--- src/asciinema/endpoint/asciicasts.clj | 21 +++++++++++++++++++++ 3 files changed, 27 insertions(+), 4 deletions(-) create mode 100644 src/asciinema/endpoint/asciicasts.clj diff --git a/project.clj b/project.clj index 3cec63c..79a7ed6 100644 --- a/project.clj +++ b/project.clj @@ -7,7 +7,9 @@ [metosin/ring-http-response "0.8.1"] [clj-time "0.12.0"] [duct "0.8.2"] - [compojure "1.5.2"] + [compojure "1.5.1"] + [metosin/compojure-api "1.1.10"] + [prismatic/schema "1.1.3"] [environ "1.1.0"] [ring "1.5.0"] [ring/ring-defaults "0.2.1"] diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index 84aa5df..0f9a1b7 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -6,12 +6,12 @@ :file-store #var asciinema.component.s3-file-store/s3-file-store :file-server #var asciinema.component.s3-file-server/s3-file-server} :endpoints - {:example #var asciinema.endpoint.example/example-endpoint} + {:asciicasts #var asciinema.endpoint.asciicasts/asciicasts-endpoint} :dependencies {:http [:app] - :app [:example] + :app [:asciicasts] :ragtime [:db] - :example [:db]} + :asciicasts [:db :file-server]} :config {:app {:middleware diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj new file mode 100644 index 0000000..8674d61 --- /dev/null +++ b/src/asciinema/endpoint/asciicasts.clj @@ -0,0 +1,21 @@ +(ns asciinema.endpoint.asciicasts + (:require [asciinema.boundary + [asciicast-database :as adb] + [file-server :as fserver]] + [asciinema.model.asciicast :as asciicast] + [compojure.api.sweet :refer :all] + [ring.util.http-response :as response] + [schema.core :as s])) + +(defn asciicasts-endpoint [{:keys [db file-server]}] + (api + (context + "/a" [] + (GET "/:token.json" [] + :path-params [token :- String] + :query-params [{dl :- s/Bool false}] + (if-let [asciicast (adb/get-asciicast-by-token db token)] + (let [path (asciicast/json-store-path asciicast) + filename (str "asciicast-" (:id asciicast) ".json")] + (fserver/serve file-server path (when dl {:filename filename}))) + (response/not-found)))))) From ce92ccd6e1b4b0db36cf61cd199d9cefc20cac21 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 14 Feb 2017 12:28:41 +0100 Subject: [PATCH 11/46] Add S3 configuration bindings --- src/asciinema/main.clj | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/asciinema/main.clj b/src/asciinema/main.clj index 459fc60..5402412 100644 --- a/src/asciinema/main.clj +++ b/src/asciinema/main.clj @@ -8,7 +8,10 @@ (defn -main [& args] (let [bindings {'http-port (Integer/parseInt (:port env "3000")) - 'db-uri (:database-url env)} + 'db-uri (:database-url env) + 's3-bucket (:s3-bucket env) + 's3-access-key (:s3-access-key env) + 's3-secret-key (:s3-secret-key env)} system (->> (load-system [(io/resource "asciinema/system.edn")] bindings) (component/start))] (add-shutdown-hook ::stop-system #(component/stop system)) From a87b76cc60b9d21cb162a0f43a446818cc796995 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 14 Feb 2017 15:33:43 +0100 Subject: [PATCH 12/46] Prevent compojure-api handle unknown exceptions --- src/asciinema/endpoint/asciicasts.clj | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index 8674d61..f43fd74 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -7,8 +7,12 @@ [ring.util.http-response :as response] [schema.core :as s])) +(defn exception-handler [^Exception e data request] + (throw e)) + (defn asciicasts-endpoint [{:keys [db file-server]}] (api + {:exceptions {:handlers {:compojure.api.exception/default exception-handler}}} (context "/a" [] (GET "/:token.json" [] From b00e66cf3c810fdd7064b06f7be5e23bc8b70531 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 14 Feb 2017 15:34:17 +0100 Subject: [PATCH 13/46] Add ring req/resp logging --- dev/resources/dev.edn | 2 +- project.clj | 1 + resources/asciinema/system.edn | 3 ++- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/dev/resources/dev.edn b/dev/resources/dev.edn index 3ee655c..594b2d4 100644 --- a/dev/resources/dev.edn +++ b/dev/resources/dev.edn @@ -7,7 +7,7 @@ {:app {:middleware {:functions {:stacktrace #var ring.middleware.stacktrace/wrap-stacktrace} - :applied ^:replace [:not-found :webjars :ring-defaults :route-aliases :stacktrace]}} + :applied ^:replace [:not-found :webjars :ring-defaults :route-aliases :ring-logger :stacktrace]}} :http {:port 4000} :db diff --git a/project.clj b/project.clj index 79a7ed6..967ab91 100644 --- a/project.clj +++ b/project.clj @@ -15,6 +15,7 @@ [ring/ring-defaults "0.2.1"] [ring-jetty-component "0.3.1"] [ring-webjars "0.1.1"] + [ring-logger-timbre "0.7.5"] [clj-aws-s3 "0.3.10" :exclusions [joda-time com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.core/jackson-annotations]] [aleph "0.4.1"] [org.slf4j/slf4j-nop "1.7.21"] diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index 0f9a1b7..a3eecaa 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -20,9 +20,10 @@ :not-found #var duct.middleware.not-found/wrap-not-found :ring-defaults #var ring.middleware.defaults/wrap-defaults :route-aliases #var duct.middleware.route-aliases/wrap-route-aliases + :ring-logger #var ring.logger.timbre/wrap-with-logger :webjars #var ring.middleware.webjars/wrap-webjars} :applied - [:not-found :webjars :ring-defaults :route-aliases :hide-errors] + [:not-found :webjars :ring-defaults :route-aliases :ring-logger :hide-errors] :arguments {:not-found #resource "asciinema/errors/404.html" :hide-errors #resource "asciinema/errors/500.html" From 0a3289f4c1c92122b00d9a53f42ee76acbde561b Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 14 Feb 2017 22:45:40 +0100 Subject: [PATCH 14/46] Report exceptions to Bugsnag --- project.clj | 1 + resources/asciinema/system.edn | 8 +++++++- src/asciinema/main.clj | 5 ++++- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/project.clj b/project.clj index 967ab91..23a3ec6 100644 --- a/project.clj +++ b/project.clj @@ -16,6 +16,7 @@ [ring-jetty-component "0.3.1"] [ring-webjars "0.1.1"] [ring-logger-timbre "0.7.5"] + [clj-bugsnag "0.2.9"] [clj-aws-s3 "0.3.10" :exclusions [joda-time com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.core/jackson-annotations]] [aleph "0.4.1"] [org.slf4j/slf4j-nop "1.7.21"] diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index a3eecaa..99b80fd 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -21,12 +21,18 @@ :ring-defaults #var ring.middleware.defaults/wrap-defaults :route-aliases #var duct.middleware.route-aliases/wrap-route-aliases :ring-logger #var ring.logger.timbre/wrap-with-logger + :bugsnag #var clj-bugsnag.ring/wrap-bugsnag :webjars #var ring.middleware.webjars/wrap-webjars} :applied - [:not-found :webjars :ring-defaults :route-aliases :ring-logger :hide-errors] + [:not-found :webjars :ring-defaults :route-aliases :ring-logger :bugsnag :hide-errors] :arguments {:not-found #resource "asciinema/errors/404.html" :hide-errors #resource "asciinema/errors/500.html" + :bugsnag + {:api-key bugsnag-key + :environment env-name + :version git-sha + :project-ns "asciinema"} :route-aliases {"/" "/index.html"} :ring-defaults {:params {:urlencoded true diff --git a/src/asciinema/main.clj b/src/asciinema/main.clj index 5402412..ae2b646 100644 --- a/src/asciinema/main.clj +++ b/src/asciinema/main.clj @@ -8,7 +8,10 @@ (defn -main [& args] (let [bindings {'http-port (Integer/parseInt (:port env "3000")) - 'db-uri (:database-url env) + 'db-uri (:database-url env) + 'env-name (:env-name env "production") + 'git-sha (:git-sha env) + 'bugsnag-key (:bugsnag-key env) 's3-bucket (:s3-bucket env) 's3-access-key (:s3-access-key env) 's3-secret-key (:s3-secret-key env)} From 597c838d35307b9b6bf88f929dfb0203bee2c738 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 15 Feb 2017 12:19:46 +0000 Subject: [PATCH 15/46] Remove comment --- src/asciinema/model/asciicast.clj | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/asciinema/model/asciicast.clj b/src/asciinema/model/asciicast.clj index 1d0cfed..a9a35b3 100644 --- a/src/asciinema/model/asciicast.clj +++ b/src/asciinema/model/asciicast.clj @@ -8,6 +8,6 @@ (def default-theme "asciinema") (defn theme-name [asciicast user] - (or (:theme_name asciicast) ; ensure we don't store empty strings in db + (or (:theme_name asciicast) (:theme_name user) default-theme)) From d5addad542a0d367dcc2219093fddb38fe6e148e Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 15 Feb 2017 12:48:35 +0000 Subject: [PATCH 16/46] Implement *Database protocols in our db component ns --- resources/asciinema/system.edn | 2 +- src/asciinema/boundary/asciicast_database.clj | 33 +----------- src/asciinema/boundary/user_database.clj | 4 ++ src/asciinema/component/db.clj | 50 +++++++++++++++++++ .../db_test.clj} | 21 ++++---- 5 files changed, 67 insertions(+), 43 deletions(-) create mode 100644 src/asciinema/boundary/user_database.clj create mode 100644 src/asciinema/component/db.clj rename test/asciinema/{boundary/asciicast_database_test.clj => component/db_test.clj} (70%) diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index 99b80fd..2016ee7 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -1,7 +1,7 @@ {:components {:app #var duct.component.handler/handler-component :http #var asciinema.component.aleph/aleph-server - :db #var duct.component.hikaricp/hikaricp + :db #var asciinema.component.db/hikaricp :ragtime #var duct.component.ragtime/ragtime :file-store #var asciinema.component.s3-file-store/s3-file-store :file-server #var asciinema.component.s3-file-server/s3-file-server} diff --git a/src/asciinema/boundary/asciicast_database.clj b/src/asciinema/boundary/asciicast_database.clj index 8fb904d..f06e5b5 100644 --- a/src/asciinema/boundary/asciicast_database.clj +++ b/src/asciinema/boundary/asciicast_database.clj @@ -1,36 +1,5 @@ -(ns asciinema.boundary.asciicast-database - (:require [clojure.java.jdbc :as jdbc] - [clj-time.coerce :as timec] - [duct.component.hikaricp :as hikaricp])) +(ns asciinema.boundary.asciicast-database) (defprotocol AsciicastDatabase (get-asciicast-by-id [this id]) (get-asciicast-by-token [this token])) - -(extend-protocol clojure.java.jdbc/ISQLValue - org.joda.time.DateTime - (sql-value [val] - (timec/to-sql-time val))) - -(extend-protocol clojure.java.jdbc/IResultSetReadColumn - java.sql.Timestamp - (result-set-read-column [x _ _] - (timec/from-sql-time x))) - -(def q-get-asciicast-by-id "SELECT * FROM asciicasts WHERE id=?") -(def q-get-asciicast-by-secret-token "SELECT * FROM asciicasts WHERE secret_token=?") -(def q-get-public-asciicast-by-id "SELECT * FROM asciicasts WHERE id=? AND private=FALSE") - -(extend-protocol AsciicastDatabase - duct.component.hikaricp.HikariCP - - (get-asciicast-by-id [{db :spec} id] - (first (jdbc/query db [q-get-asciicast-by-id id]))) - - (get-asciicast-by-token [{db :spec} token] - (when-let [query (cond - (re-matches #"\d+" token) - [q-get-public-asciicast-by-id (Long/parseLong token)] - (= (count token) 25) - [q-get-asciicast-by-secret-token token])] - (first (jdbc/query db query))))) diff --git a/src/asciinema/boundary/user_database.clj b/src/asciinema/boundary/user_database.clj new file mode 100644 index 0000000..abae4aa --- /dev/null +++ b/src/asciinema/boundary/user_database.clj @@ -0,0 +1,4 @@ +(ns asciinema.boundary.user-database) + +(defprotocol UserDatabase + (get-user-by-id [this id])) diff --git a/src/asciinema/component/db.clj b/src/asciinema/component/db.clj new file mode 100644 index 0000000..85bf0e5 --- /dev/null +++ b/src/asciinema/component/db.clj @@ -0,0 +1,50 @@ +(ns asciinema.component.db + (:require [asciinema.boundary.asciicast-database :refer :all] + [asciinema.boundary.user-database :refer :all] + [clojure.java.jdbc :as jdbc] + [clj-time.coerce :as timec] + [duct.component.hikaricp :as hikaricp])) + +(extend-protocol jdbc/ISQLValue + org.joda.time.DateTime + (sql-value [val] + (timec/to-sql-time val))) + +(extend-protocol jdbc/IResultSetReadColumn + java.sql.Timestamp + (result-set-read-column [x _ _] + (timec/from-sql-time x))) + +;; AsciicastDatabase + +(def q-get-asciicast-by-id "SELECT * FROM asciicasts WHERE id=?") +(def q-get-asciicast-by-secret-token "SELECT * FROM asciicasts WHERE secret_token=?") +(def q-get-public-asciicast-by-id "SELECT * FROM asciicasts WHERE id=? AND private=FALSE") + +(extend-protocol AsciicastDatabase + duct.component.hikaricp.HikariCP + + (get-asciicast-by-id [{db :spec} id] + (first (jdbc/query db [q-get-asciicast-by-id id]))) + + (get-asciicast-by-token [{db :spec} token] + (when-let [query (cond + (re-matches #"\d+" token) + [q-get-public-asciicast-by-id (Long/parseLong token)] + (= (count token) 25) + [q-get-asciicast-by-secret-token token])] + (first (jdbc/query db query))))) + +;; UserDatabase + +(def q-get-user-by-id "SELECT * FROM users WHERE id=?") + +(extend-protocol UserDatabase + duct.component.hikaricp.HikariCP + + (get-user-by-id [{db :spec} id] + (first (jdbc/query db [q-get-user-by-id id])))) + +;; constructor + +(def hikaricp hikaricp/hikaricp) diff --git a/test/asciinema/boundary/asciicast_database_test.clj b/test/asciinema/component/db_test.clj similarity index 70% rename from test/asciinema/boundary/asciicast_database_test.clj rename to test/asciinema/component/db_test.clj index 5860bad..4a5fba7 100644 --- a/test/asciinema/boundary/asciicast_database_test.clj +++ b/test/asciinema/component/db_test.clj @@ -1,12 +1,13 @@ -(ns asciinema.boundary.asciicast-database-test +(ns asciinema.component.db-test (:require [clojure.test :refer :all] [clojure.java.jdbc :as jdbc] [clj-time.local :as timel] [com.stuartsierra.component :as component] - [asciinema.boundary.asciicast-database :as db])) + [asciinema.component.db :as db] + [asciinema.boundary.asciicast-database :as adb])) (defmacro with-db-component [component-var & body] - `(let [component# (-> (duct.component.hikaricp/hikaricp {:uri "jdbc:postgresql://localhost:15432/asciinema_test?user=vagrant"}) + `(let [component# (-> (db/hikaricp {:uri "jdbc:postgresql://localhost:15432/asciinema_test?user=vagrant"}) component/start)] (try (jdbc/with-db-transaction [db# (:spec component#)] @@ -32,22 +33,22 @@ (testing "for existing asciicast" (with-db-component db (let [asciicast (insert-asciicast (:spec db))] - (is (map? (db/get-asciicast-by-id db (:id asciicast))))))) + (is (map? (adb/get-asciicast-by-id db (:id asciicast))))))) (testing "for non-existing asciicast" (with-db-component db - (is (nil? (db/get-asciicast-by-id db 1)))))) + (is (nil? (adb/get-asciicast-by-id db 1)))))) (deftest get-asciicast-by-token-test (testing "for existing public asciicast" (with-db-component db (let [asciicast (insert-asciicast (:spec db) {:private false})] - (is (map? (db/get-asciicast-by-token db (:secret_token asciicast)))) - (is (map? (db/get-asciicast-by-token db (-> asciicast :id str))))))) + (is (map? (adb/get-asciicast-by-token db (:secret_token asciicast)))) + (is (map? (adb/get-asciicast-by-token db (-> asciicast :id str))))))) (testing "for existing private asciicast" (with-db-component db (let [asciicast (insert-asciicast (:spec db) {:private true})] - (is (map? (db/get-asciicast-by-token db (:secret_token asciicast)))) - (is (nil? (db/get-asciicast-by-token db (-> asciicast :id str))))))) + (is (map? (adb/get-asciicast-by-token db (:secret_token asciicast)))) + (is (nil? (adb/get-asciicast-by-token db (-> asciicast :id str))))))) (testing "for non-existing asciicast" (with-db-component db - (is (nil? (db/get-asciicast-by-token db "1")))))) + (is (nil? (adb/get-asciicast-by-token db "1")))))) From 6943bceccab639f5f71815b078299664f499a6d7 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 15 Feb 2017 16:53:24 +0100 Subject: [PATCH 17/46] Remove example endpoint --- src/asciinema/endpoint/example.clj | 8 -------- test/asciinema/endpoint/example_test.clj | 16 ---------------- 2 files changed, 24 deletions(-) delete mode 100644 src/asciinema/endpoint/example.clj delete mode 100644 test/asciinema/endpoint/example_test.clj diff --git a/src/asciinema/endpoint/example.clj b/src/asciinema/endpoint/example.clj deleted file mode 100644 index 46c9821..0000000 --- a/src/asciinema/endpoint/example.clj +++ /dev/null @@ -1,8 +0,0 @@ -(ns asciinema.endpoint.example - (:require [compojure.core :refer :all] - [clojure.java.io :as io])) - -(defn example-endpoint [{{db :spec} :db}] - (context "/example" [] - (GET "/" [] - (io/resource "asciinema/endpoint/example/example.html")))) diff --git a/test/asciinema/endpoint/example_test.clj b/test/asciinema/endpoint/example_test.clj deleted file mode 100644 index ec32ab1..0000000 --- a/test/asciinema/endpoint/example_test.clj +++ /dev/null @@ -1,16 +0,0 @@ -(ns asciinema.endpoint.example-test - (:require [com.stuartsierra.component :as component] - [clojure.test :refer :all] - [kerodon.core :refer :all] - [kerodon.test :refer :all] - [shrubbery.core :as shrub] - [asciinema.endpoint.example :as example])) - -(def handler - (example/example-endpoint {})) - -(deftest smoke-test - (testing "example page exists" - (-> (session handler) - (visit "/example") - (has (status? 200) "page exists")))) From 2e0cb5c3ebcc4b268d95d0518bf029892cf2c1ab Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 15 Feb 2017 17:53:19 +0100 Subject: [PATCH 18/46] Unify FileStore and FileServer --- dev/resources/dev.edn | 5 +-- dev/src/dev.clj | 4 +- resources/asciinema/system.edn | 10 +---- src/asciinema/boundary/file_server.clj | 4 -- src/asciinema/boundary/file_store.clj | 3 +- src/asciinema/component/local_file_server.clj | 18 --------- src/asciinema/component/local_file_store.clj | 16 +++++++- src/asciinema/component/s3_file_server.clj | 37 ------------------- src/asciinema/component/s3_file_store.clj | 37 ++++++++++++++++++- src/asciinema/endpoint/asciicasts.clj | 6 +-- test/asciinema/boundary/file_server_test.clj | 7 ---- 11 files changed, 59 insertions(+), 88 deletions(-) delete mode 100644 src/asciinema/boundary/file_server.clj delete mode 100644 src/asciinema/component/local_file_server.clj delete mode 100644 src/asciinema/component/s3_file_server.clj delete mode 100644 test/asciinema/boundary/file_server_test.clj diff --git a/dev/resources/dev.edn b/dev/resources/dev.edn index 594b2d4..8173efd 100644 --- a/dev/resources/dev.edn +++ b/dev/resources/dev.edn @@ -1,8 +1,5 @@ {:components - {:file-store #var asciinema.component.local-file-store/local-file-store - :file-server #var asciinema.component.local-file-server/local-file-server} - :dependencies - {:file-server [:file-store]} + {:file-store #var asciinema.component.local-file-store/local-file-store} :config {:app {:middleware diff --git a/dev/src/dev.clj b/dev/src/dev.clj index b720e61..7faf3cf 100644 --- a/dev/src/dev.clj +++ b/dev/src/dev.clj @@ -9,10 +9,10 @@ [duct.util.repl :refer [setup test cljs-repl migrate rollback]] [duct.util.system :refer [load-system]] [reloaded.repl :refer [system init start stop go reset]] - [asciinema.boundary.file-server :as file-server] [asciinema.boundary.file-store :as file-store] + [asciinema.boundary.asciicast-database :as asciicast-database] [asciinema.component.local-file-store :refer [->LocalFileStore]] - [asciinema.component.local-file-server :refer [->LocalFileServer]])) + [asciinema.component.s3-file-store :refer [->S3FileStore]])) (defn new-system [] (load-system (keep io/resource ["asciinema/system.edn" "dev.edn" "local.edn"]))) diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index 2016ee7..e4ba3a6 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -3,15 +3,14 @@ :http #var asciinema.component.aleph/aleph-server :db #var asciinema.component.db/hikaricp :ragtime #var duct.component.ragtime/ragtime - :file-store #var asciinema.component.s3-file-store/s3-file-store - :file-server #var asciinema.component.s3-file-server/s3-file-server} + :file-store #var asciinema.component.s3-file-store/s3-file-store} :endpoints {:asciicasts #var asciinema.endpoint.asciicasts/asciicasts-endpoint} :dependencies {:http [:app] :app [:asciicasts] :ragtime [:db] - :asciicasts [:db :file-server]} + :asciicasts [:db :file-store]} :config {:app {:middleware @@ -58,11 +57,6 @@ :ragtime {:resource-path "asciinema/migrations"} :file-store - {:cred {:access-key s3-access-key - :secret-key s3-secret-key} - :bucket s3-bucket - :path-prefix "uploads/"} - :file-server {:cred {:access-key s3-access-key :secret-key s3-secret-key} :bucket s3-bucket diff --git a/src/asciinema/boundary/file_server.clj b/src/asciinema/boundary/file_server.clj deleted file mode 100644 index c380724..0000000 --- a/src/asciinema/boundary/file_server.clj +++ /dev/null @@ -1,4 +0,0 @@ -(ns asciinema.boundary.file-server) - -(defprotocol FileServer - (serve [this path] [this path opts])) diff --git a/src/asciinema/boundary/file_store.clj b/src/asciinema/boundary/file_store.clj index d9107a8..c9fad8c 100644 --- a/src/asciinema/boundary/file_store.clj +++ b/src/asciinema/boundary/file_store.clj @@ -4,4 +4,5 @@ (put-file [this file path] [this file path size]) (input-stream [this path]) (move-file [this old-path new-path]) - (delete-file [this path])) + (delete-file [this path]) + (serve-file [this path opts])) diff --git a/src/asciinema/component/local_file_server.clj b/src/asciinema/component/local_file_server.clj deleted file mode 100644 index eeab31d..0000000 --- a/src/asciinema/component/local_file_server.clj +++ /dev/null @@ -1,18 +0,0 @@ -(ns asciinema.component.local-file-server - (:require [asciinema.boundary - [file-server :as file-server] - [file-store :as file-store]] - [ring.util.http-response :as response])) - -(defrecord LocalFileServer [file-store] - file-server/FileServer - (serve [this path] - (file-server/serve this path {})) - (serve [this path {:keys [filename]}] - (let [resp (response/ok (file-store/input-stream file-store path))] - (if filename - (response/header resp "Content-Disposition" (str "attachment; filename=" filename)) - resp)))) - -(defn local-file-server [{:keys [file-store]}] - (->LocalFileServer file-store)) diff --git a/src/asciinema/component/local_file_store.clj b/src/asciinema/component/local_file_store.clj index e250f1d..21823a0 100644 --- a/src/asciinema/component/local_file_store.clj +++ b/src/asciinema/component/local_file_store.clj @@ -1,25 +1,37 @@ (ns asciinema.component.local-file-store (:require [asciinema.boundary.file-store :as file-store] - [clojure.java.io :as io])) + [clojure.java.io :as io] + [ring.util.http-response :as response])) (defrecord LocalFileStore [base-path] file-store/FileStore + (put-file [this file path] (let [path (str base-path path)] (io/make-parents path) (io/copy file (io/file path)))) + (put-file [this file path size] (file-store/put-file this file path)) + (input-stream [this path] (let [path (str base-path path)] (io/input-stream path))) + (move-file [this old-path new-path] (let [old-path (str base-path old-path) new-path (str base-path new-path)] (.renameTo (io/file old-path) (io/file new-path)))) + (delete-file [this path] (let [path (str base-path path)] - (io/delete-file path)))) + (io/delete-file path))) + + (serve-file [this path {:keys [filename]}] + (let [resp (response/ok (file-store/input-stream this path))] + (if filename + (response/header resp "Content-Disposition" (str "attachment; filename=" filename)) + resp)))) (defn local-file-store [{:keys [path]}] (->LocalFileStore path)) diff --git a/src/asciinema/component/s3_file_server.clj b/src/asciinema/component/s3_file_server.clj deleted file mode 100644 index 2c04a39..0000000 --- a/src/asciinema/component/s3_file_server.clj +++ /dev/null @@ -1,37 +0,0 @@ -(ns asciinema.component.s3-file-server - (:require [asciinema.boundary.file-server :as file-server] - [clj-time - [coerce :as timec] - [core :as time]] - [ring.util.http-response :as response]) - (:import com.amazonaws.auth.BasicAWSCredentials - com.amazonaws.services.s3.AmazonS3Client - [com.amazonaws.services.s3.model GeneratePresignedUrlRequest ResponseHeaderOverrides])) - -(defn- s3-client* [cred] - (let [credentials (BasicAWSCredentials. (:access-key cred) (:secret-key cred))] - (AmazonS3Client. credentials))) - -(def ^:private s3-client (memoize s3-client*)) - -(defn- generate-presigned-url [cred bucket path {:keys [expires filename] - :or {expires (-> 1 time/days time/from-now)}}] - (let [client (s3-client cred) - request (GeneratePresignedUrlRequest. bucket path)] - (.setExpiration request (timec/to-date expires)) - (when filename - (let [header-overrides (doto (ResponseHeaderOverrides.) - (.setContentDisposition (str "attachment; filename=" filename)))] - (.setResponseHeaders request header-overrides))) - (.toString (.generatePresignedUrl client request)))) - -(defrecord S3FileServer [cred bucket path-prefix] - file-server/FileServer - (serve [this path] - (file-server/serve this path {})) - (serve [this path opts] - (let [path (str path-prefix path)] - (response/found (generate-presigned-url cred bucket path opts))))) - -(defn s3-file-server [{:keys [cred bucket path-prefix]}] - (->S3FileServer cred bucket path-prefix)) diff --git a/src/asciinema/component/s3_file_store.clj b/src/asciinema/component/s3_file_store.clj index e9fb608..4013f49 100644 --- a/src/asciinema/component/s3_file_store.clj +++ b/src/asciinema/component/s3_file_store.clj @@ -1,25 +1,58 @@ (ns asciinema.component.s3-file-store (:require [asciinema.boundary.file-store :as file-store] - [aws.sdk.s3 :as s3])) + [aws.sdk.s3 :as s3] + [clj-time + [coerce :as timec] + [core :as time]] + [ring.util.http-response :as response]) + (:import com.amazonaws.auth.BasicAWSCredentials + com.amazonaws.services.s3.AmazonS3Client + [com.amazonaws.services.s3.model GeneratePresignedUrlRequest ResponseHeaderOverrides])) + +(defn- s3-client* [cred] + (let [credentials (BasicAWSCredentials. (:access-key cred) (:secret-key cred))] + (AmazonS3Client. credentials))) + +(def ^:private s3-client (memoize s3-client*)) + +(defn- generate-presigned-url [cred bucket path {:keys [expires filename] + :or {expires (-> 1 time/days time/from-now)}}] + (let [client (s3-client cred) + request (GeneratePresignedUrlRequest. bucket path)] + (.setExpiration request (timec/to-date expires)) + (when filename + (let [header-overrides (doto (ResponseHeaderOverrides.) + (.setContentDisposition (str "attachment; filename=" filename)))] + (.setResponseHeaders request header-overrides))) + (.toString (.generatePresignedUrl client request)))) (defrecord S3FileStore [cred bucket path-prefix] file-store/FileStore + (put-file [this file path] (file-store/put-file this file path nil)) + (put-file [this file path size] (let [path (str path-prefix path)] (s3/put-object cred bucket path file {:content-length size}))) + (input-stream [this path] (let [path (str path-prefix path)] (:content (s3/get-object cred bucket path)))) + (move-file [this old-path new-path] (let [old-path (str path-prefix old-path) new-path (str path-prefix new-path)] (s3/copy-object cred bucket old-path new-path) (s3/delete-object cred bucket old-path))) + (delete-file [this path] (let [path (str path-prefix path)] - (s3/delete-object cred bucket path)))) + (s3/delete-object cred bucket path))) + + (serve-file [this path opts] + (let [path (str path-prefix path)] + (response/found (generate-presigned-url cred bucket path opts))))) (defn s3-file-store [{:keys [cred bucket path-prefix]}] (->S3FileStore cred bucket path-prefix)) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index f43fd74..c1b17f2 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -1,7 +1,7 @@ (ns asciinema.endpoint.asciicasts (:require [asciinema.boundary [asciicast-database :as adb] - [file-server :as fserver]] + [file-store :as fstore]] [asciinema.model.asciicast :as asciicast] [compojure.api.sweet :refer :all] [ring.util.http-response :as response] @@ -10,7 +10,7 @@ (defn exception-handler [^Exception e data request] (throw e)) -(defn asciicasts-endpoint [{:keys [db file-server]}] +(defn asciicasts-endpoint [{:keys [db file-store]}] (api {:exceptions {:handlers {:compojure.api.exception/default exception-handler}}} (context @@ -21,5 +21,5 @@ (if-let [asciicast (adb/get-asciicast-by-token db token)] (let [path (asciicast/json-store-path asciicast) filename (str "asciicast-" (:id asciicast) ".json")] - (fserver/serve file-server path (when dl {:filename filename}))) + (fstore/serve-file file-store path (when dl {:filename filename}))) (response/not-found)))))) diff --git a/test/asciinema/boundary/file_server_test.clj b/test/asciinema/boundary/file_server_test.clj deleted file mode 100644 index 2136484..0000000 --- a/test/asciinema/boundary/file_server_test.clj +++ /dev/null @@ -1,7 +0,0 @@ -(ns asciinema.boundary.file-server-test - (:require [clojure.test :refer :all] - [asciinema.boundary.file-server :as file-server])) - -(deftest a-test - (testing "FIXME, I fail." - (is (= 0 1)))) From f18461643285533aebdecba004ec69aacf174a50 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Thu, 16 Feb 2017 13:44:49 +0100 Subject: [PATCH 19/46] PNG generation --- project.clj | 1 + src/asciinema/endpoint/asciicasts.clj | 51 +++++++++++++++++++++++++-- src/asciinema/model/asciicast.clj | 27 +++++++++++++- src/asciinema/util/io.clj | 15 ++++++++ 4 files changed, 91 insertions(+), 3 deletions(-) create mode 100644 src/asciinema/util/io.clj diff --git a/project.clj b/project.clj index 23a3ec6..2a6931c 100644 --- a/project.clj +++ b/project.clj @@ -19,6 +19,7 @@ [clj-bugsnag "0.2.9"] [clj-aws-s3 "0.3.10" :exclusions [joda-time com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.core/jackson-annotations]] [aleph "0.4.1"] + [pandect "0.6.1"] [org.slf4j/slf4j-nop "1.7.21"] [org.webjars/normalize.css "3.0.2"] [duct/hikaricp-component "0.1.0"] diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index c1b17f2..c3d5f7f 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -1,15 +1,38 @@ (ns asciinema.endpoint.asciicasts (:require [asciinema.boundary [asciicast-database :as adb] - [file-store :as fstore]] + [file-store :as fstore] + [user-database :as udb]] [asciinema.model.asciicast :as asciicast] + [asciinema.util.io :refer [with-tmp-dir]] + [clojure.java.io :as io] + [clojure.java.shell :as shell] [compojure.api.sweet :refer :all] + [environ.core :refer [env]] [ring.util.http-response :as response] - [schema.core :as s])) + [schema.core :as s] + [clojure.string :as str])) (defn exception-handler [^Exception e data request] (throw e)) +(defn a2png [in-url out-path {:keys [snapshot-at theme scale]}] + (let [a2png-bin (:a2png-bin env "a2png/a2png.sh") + {:keys [exit] :as result} (shell/sh a2png-bin + "-t" theme + "-s" (str scale) + in-url + out-path + (str snapshot-at))] + (when-not (zero? exit) + (throw (ex-info "a2png error" result))))) + +(def Num (s/if #(str/includes? % ".") + Double + s/Int)) + +(def Theme (apply s/enum asciicast/themes)) + (defn asciicasts-endpoint [{:keys [db file-store]}] (api {:exceptions {:handlers {:compojure.api.exception/default exception-handler}}} @@ -22,4 +45,28 @@ (let [path (asciicast/json-store-path asciicast) filename (str "asciicast-" (:id asciicast) ".json")] (fstore/serve-file file-store path (when dl {:filename filename}))) + (response/not-found))) + + (GET "/:token.png" [] + :path-params [token :- String] + :query-params [{time :- Num nil} + {theme :- Theme nil} + {scale :- (s/enum "1" "2") nil}] + (if-let [asciicast (adb/get-asciicast-by-token db token)] + (let [user (udb/get-user-by-id db (:user_id asciicast)) + png-params (cond-> (asciicast/png-params asciicast user) + time (assoc :snapshot-at time) + theme (assoc :theme theme) + scale (assoc :scale (Integer/parseInt scale))) + json-store-path (asciicast/json-store-path asciicast) + png-store-path (asciicast/png-store-path asciicast png-params)] + (with-tmp-dir [dir "asciinema-png-"] + (let [json-local-path (str dir "/asciicast.json") + png-local-path (str dir "/asciicast.png")] + (with-open [in (fstore/input-stream file-store json-store-path)] + (let [out (io/file json-local-path)] + (io/copy in out))) + (a2png json-local-path png-local-path png-params) + (fstore/put-file file-store (io/file png-local-path) png-store-path))) + (fstore/serve-file file-store png-store-path {})) (response/not-found)))))) diff --git a/src/asciinema/model/asciicast.clj b/src/asciinema/model/asciicast.clj index a9a35b3..bd2bc19 100644 --- a/src/asciinema/model/asciicast.clj +++ b/src/asciinema/model/asciicast.clj @@ -1,13 +1,38 @@ -(ns asciinema.model.asciicast) +(ns asciinema.model.asciicast + (:require[pandect.algo.sha1 :as sha1] + [clojure.string :as str])) (defn json-store-path [{:keys [id file stdout_frames]}] (cond file (str "asciicast/file/" id "/" file) stdout_frames (str "asciicast/stdout_frames/" id "/" stdout_frames))) +(def themes #{"asciinema" "tango" "solarized-dark" "solarized-light" "monokai"}) (def default-theme "asciinema") (defn theme-name [asciicast user] (or (:theme_name asciicast) (:theme_name user) default-theme)) + +(defn snapshot-at [{:keys [snapshot_at duration]}] + (or snapshot_at (/ duration 2.0))) + +(def default-png-scale 2) + +(defn png-params [asciicast user] + {:snapshot-at (snapshot-at asciicast) + :theme (theme-name asciicast user) + :scale default-png-scale}) + +(defn- png-version [asciicast params] + (let [attrs (assoc params :id (:id asciicast))] + (->> attrs + (map (fn [[k v]] (str (name k) "=" v))) + (str/join "/") + (sha1/sha1)))) + +(defn png-store-path [asciicast params] + (let [ver (png-version asciicast params) + png-filename (str ver ".png")] + (str "png/" (:id asciicast) "/" png-filename))) diff --git a/src/asciinema/util/io.clj b/src/asciinema/util/io.clj new file mode 100644 index 0000000..cdc4676 --- /dev/null +++ b/src/asciinema/util/io.clj @@ -0,0 +1,15 @@ +(ns asciinema.util.io + (:require [clojure.java.shell :as shell]) + (:import java.nio.file.Files + java.nio.file.attribute.FileAttribute)) + +(defn create-tmp-dir [prefix] + (let [dir (Files/createTempDirectory prefix (into-array FileAttribute []))] + (.toFile dir))) + +(defmacro with-tmp-dir [[sym prefix] & body] + `(let [~sym (create-tmp-dir ~prefix)] + (try + ~@body + (finally + (shell/sh "rm" "-rf" (.getPath ~sym)))))) From 368d0787cfb7eba15c77e0b621298a24f0aa5a26 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Thu, 16 Feb 2017 16:41:30 +0100 Subject: [PATCH 20/46] Auto-expire png files after 7 days --- dev/resources/dev.edn | 3 +- project.clj | 1 + resources/asciinema/system.edn | 10 +++++-- src/asciinema/boundary/expiring_set.clj | 6 ++++ src/asciinema/component/mem_expiring_set.clj | 14 +++++++++ src/asciinema/component/redis_client.clj | 28 ++++++++++++++++++ src/asciinema/endpoint/asciicasts.clj | 31 ++++++++++++-------- src/asciinema/main.clj | 4 ++- 8 files changed, 80 insertions(+), 17 deletions(-) create mode 100644 src/asciinema/boundary/expiring_set.clj create mode 100644 src/asciinema/component/mem_expiring_set.clj create mode 100644 src/asciinema/component/redis_client.clj diff --git a/dev/resources/dev.edn b/dev/resources/dev.edn index 8173efd..4733fa0 100644 --- a/dev/resources/dev.edn +++ b/dev/resources/dev.edn @@ -1,5 +1,6 @@ {:components - {:file-store #var asciinema.component.local-file-store/local-file-store} + {:file-store #var asciinema.component.local-file-store/local-file-store + :exp-set #var asciinema.component.mem-expiring-set/mem-expiring-set} :config {:app {:middleware diff --git a/project.clj b/project.clj index 2a6931c..c7e20c2 100644 --- a/project.clj +++ b/project.clj @@ -20,6 +20,7 @@ [clj-aws-s3 "0.3.10" :exclusions [joda-time com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.core/jackson-annotations]] [aleph "0.4.1"] [pandect "0.6.1"] + [com.taoensso/carmine "2.15.1"] [org.slf4j/slf4j-nop "1.7.21"] [org.webjars/normalize.css "3.0.2"] [duct/hikaricp-component "0.1.0"] diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index e4ba3a6..3614a53 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -3,14 +3,15 @@ :http #var asciinema.component.aleph/aleph-server :db #var asciinema.component.db/hikaricp :ragtime #var duct.component.ragtime/ragtime - :file-store #var asciinema.component.s3-file-store/s3-file-store} + :file-store #var asciinema.component.s3-file-store/s3-file-store + :exp-set #var asciinema.component.redis-client/redis-client} :endpoints {:asciicasts #var asciinema.endpoint.asciicasts/asciicasts-endpoint} :dependencies {:http [:app] :app [:asciicasts] :ragtime [:db] - :asciicasts [:db :file-store]} + :asciicasts [:db :file-store :exp-set]} :config {:app {:middleware @@ -60,4 +61,7 @@ {:cred {:access-key s3-access-key :secret-key s3-secret-key} :bucket s3-bucket - :path-prefix "uploads/"}}} + :path-prefix "uploads/"} + :exp-set + {:host redis-host + :port redis-port}}} diff --git a/src/asciinema/boundary/expiring_set.clj b/src/asciinema/boundary/expiring_set.clj new file mode 100644 index 0000000..f17fdcd --- /dev/null +++ b/src/asciinema/boundary/expiring_set.clj @@ -0,0 +1,6 @@ +(ns asciinema.boundary.expiring-set + (:refer-clojure :exclude [conj! contains?])) + +(defprotocol ExpiringSet + (conj! [this value expires-at]) + (contains? [this value])) diff --git a/src/asciinema/component/mem_expiring_set.clj b/src/asciinema/component/mem_expiring_set.clj new file mode 100644 index 0000000..367d549 --- /dev/null +++ b/src/asciinema/component/mem_expiring_set.clj @@ -0,0 +1,14 @@ +(ns asciinema.component.mem-expiring-set + (:require [asciinema.boundary.expiring-set :as exp-set])) + +(defrecord MemExpiringSet [store] + exp-set/ExpiringSet + + (conj! [this value _expires-at] + (swap! store conj value)) + + (contains? [this value] + (contains? @store value))) + +(defn mem-expiring-set [{:keys [store]}] + (->MemExpiringSet (or store (atom #{})))) diff --git a/src/asciinema/component/redis_client.clj b/src/asciinema/component/redis_client.clj new file mode 100644 index 0000000..07af0f6 --- /dev/null +++ b/src/asciinema/component/redis_client.clj @@ -0,0 +1,28 @@ +(ns asciinema.component.redis-client + (:require [asciinema.boundary.expiring-set :as exp-set] + [clj-time.core :as t] + [clj-time.local :as tl] + [com.stuartsierra.component :as component] + [taoensso.carmine :as car])) + +(defrecord RedisClient [host port] + component/Lifecycle + (start [component] + (if (:listener component) + component + (let [conn {:pool {} :spec {:host host :port port}}] + (assoc component :conn conn)))) + (stop [component] + (if (:conn component) + (dissoc component :conn) + component)) + + exp-set/ExpiringSet + (conj! [this value expires-at] + (let [seconds (t/in-seconds (t/interval (tl/local-now) expires-at))] + (car/wcar (:conn this) (car/setex value seconds true)))) + (contains? [this value] + (car/as-bool (car/wcar (:conn this) (car/exists value))))) + +(defn redis-client [{:keys [host port]}] + (->RedisClient host port)) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index c3d5f7f..70af37d 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -1,17 +1,19 @@ (ns asciinema.endpoint.asciicasts (:require [asciinema.boundary [asciicast-database :as adb] + [expiring-set :as exp-set] [file-store :as fstore] [user-database :as udb]] [asciinema.model.asciicast :as asciicast] [asciinema.util.io :refer [with-tmp-dir]] + [clj-time.core :as t] [clojure.java.io :as io] [clojure.java.shell :as shell] + [clojure.string :as str] [compojure.api.sweet :refer :all] [environ.core :refer [env]] [ring.util.http-response :as response] - [schema.core :as s] - [clojure.string :as str])) + [schema.core :as s])) (defn exception-handler [^Exception e data request] (throw e)) @@ -33,7 +35,9 @@ (def Theme (apply s/enum asciicast/themes)) -(defn asciicasts-endpoint [{:keys [db file-store]}] +(def png-ttl-days 7) + +(defn asciicasts-endpoint [{:keys [db file-store exp-set]}] (api {:exceptions {:handlers {:compojure.api.exception/default exception-handler}}} (context @@ -58,15 +62,18 @@ time (assoc :snapshot-at time) theme (assoc :theme theme) scale (assoc :scale (Integer/parseInt scale))) - json-store-path (asciicast/json-store-path asciicast) png-store-path (asciicast/png-store-path asciicast png-params)] - (with-tmp-dir [dir "asciinema-png-"] - (let [json-local-path (str dir "/asciicast.json") - png-local-path (str dir "/asciicast.png")] - (with-open [in (fstore/input-stream file-store json-store-path)] - (let [out (io/file json-local-path)] - (io/copy in out))) - (a2png json-local-path png-local-path png-params) - (fstore/put-file file-store (io/file png-local-path) png-store-path))) + (when-not (exp-set/contains? exp-set png-store-path) + (with-tmp-dir [dir "asciinema-png-"] + (let [json-store-path (asciicast/json-store-path asciicast) + json-local-path (str dir "/asciicast.json") + png-local-path (str dir "/asciicast.png") + expires (-> png-ttl-days t/days t/from-now)] + (with-open [in (fstore/input-stream file-store json-store-path)] + (let [out (io/file json-local-path)] + (io/copy in out))) + (a2png json-local-path png-local-path png-params) + (fstore/put-file file-store (io/file png-local-path) png-store-path) + (exp-set/conj! exp-set png-store-path expires)))) (fstore/serve-file file-store png-store-path {})) (response/not-found)))))) diff --git a/src/asciinema/main.clj b/src/asciinema/main.clj index ae2b646..0a84342 100644 --- a/src/asciinema/main.clj +++ b/src/asciinema/main.clj @@ -14,7 +14,9 @@ 'bugsnag-key (:bugsnag-key env) 's3-bucket (:s3-bucket env) 's3-access-key (:s3-access-key env) - 's3-secret-key (:s3-secret-key env)} + 's3-secret-key (:s3-secret-key env) + 'redis-host (:redis-host env "localhost") + 'redis-port (Integer/parseInt (:redis-port env "6379"))} system (->> (load-system [(io/resource "asciinema/system.edn")] bindings) (component/start))] (add-shutdown-hook ::stop-system #(component/stop system)) From f0c25626d924e623762456ee45d3cae377cdb02e Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Thu, 23 Feb 2017 11:57:56 +0100 Subject: [PATCH 21/46] Update deps --- project.clj | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/project.clj b/project.clj index c7e20c2..0f74029 100644 --- a/project.clj +++ b/project.clj @@ -4,8 +4,8 @@ :min-lein-version "2.0.0" :dependencies [[org.clojure/clojure "1.8.0"] [com.stuartsierra/component "0.3.1"] - [metosin/ring-http-response "0.8.1"] - [clj-time "0.12.0"] + [metosin/ring-http-response "0.8.0"] + [clj-time "0.13.0"] [duct "0.8.2"] [compojure "1.5.1"] [metosin/compojure-api "1.1.10"] From 93ec479bb0599a5d3dcaa2593af00e1b1bc51a37 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sun, 26 Feb 2017 11:38:24 +0100 Subject: [PATCH 22/46] Generate PNG files on worker pool --- resources/asciinema/system.edn | 10 +++-- src/asciinema/boundary/executor.clj | 4 ++ .../component/fixed_thread_executor.clj | 39 +++++++++++++++++++ src/asciinema/endpoint/asciicasts.clj | 31 ++++++++------- 4 files changed, 67 insertions(+), 17 deletions(-) create mode 100644 src/asciinema/boundary/executor.clj create mode 100644 src/asciinema/component/fixed_thread_executor.clj diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index 3614a53..eaebc66 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -4,14 +4,15 @@ :db #var asciinema.component.db/hikaricp :ragtime #var duct.component.ragtime/ragtime :file-store #var asciinema.component.s3-file-store/s3-file-store - :exp-set #var asciinema.component.redis-client/redis-client} + :exp-set #var asciinema.component.redis-client/redis-client + :executor #var asciinema.component.fixed-thread-executor/fixed-thread-executor} :endpoints {:asciicasts #var asciinema.endpoint.asciicasts/asciicasts-endpoint} :dependencies {:http [:app] :app [:asciicasts] :ragtime [:db] - :asciicasts [:db :file-store :exp-set]} + :asciicasts [:db :file-store :exp-set :executor]} :config {:app {:middleware @@ -64,4 +65,7 @@ :path-prefix "uploads/"} :exp-set {:host redis-host - :port redis-port}}} + :port redis-port} + :executor + {:threads 1 + :queue-length 1}}} diff --git a/src/asciinema/boundary/executor.clj b/src/asciinema/boundary/executor.clj new file mode 100644 index 0000000..51360a1 --- /dev/null +++ b/src/asciinema/boundary/executor.clj @@ -0,0 +1,4 @@ +(ns asciinema.boundary.executor) + +(defprotocol Executor + (execute [this f])) diff --git a/src/asciinema/component/fixed_thread_executor.clj b/src/asciinema/component/fixed_thread_executor.clj new file mode 100644 index 0000000..fb7255a --- /dev/null +++ b/src/asciinema/component/fixed_thread_executor.clj @@ -0,0 +1,39 @@ +(ns asciinema.component.fixed-thread-executor + (:require [aleph.flow :as flow] + [asciinema.boundary.executor :as executor] + [com.stuartsierra.component :as component] + [manifold.deferred :as d]) + (:import [java.util.concurrent + ExecutorService + RejectedExecutionException + TimeUnit])) + +(defrecord FixedThreadExecutor [threads queue-length] + executor/Executor + (execute [{:keys [^ExecutorService executor]} f] + (try + (let [result (d/deferred) + f (fn [] + (try + (d/success! result (f)) + (catch Exception e + (d/error! result e))))] + (.execute executor f) + result) + (catch RejectedExecutionException _ + {:status 503 :headers {"Retry-After" "5"} :body "

503

"}))) + + component/Lifecycle + (start [{:keys [threads queue-length] :as component}] + (let [executor (flow/fixed-thread-executor threads {:onto? false + :initial-thread-count threads + :queue-length queue-length})] + (assoc component :executor executor))) + (stop [{:keys [^ExecutorService executor] :as component}] + (.shutdown executor) + (when-not (.awaitTermination executor 1000 TimeUnit/MILLISECONDS) + (.shutdownNow executor)) + (assoc component :executor nil))) + +(defn fixed-thread-executor [{:keys [threads queue-length]}] + (->FixedThreadExecutor threads queue-length)) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index 70af37d..eec30e4 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -3,6 +3,7 @@ [asciicast-database :as adb] [expiring-set :as exp-set] [file-store :as fstore] + [executor :as executor] [user-database :as udb]] [asciinema.model.asciicast :as asciicast] [asciinema.util.io :refer [with-tmp-dir]] @@ -37,7 +38,7 @@ (def png-ttl-days 7) -(defn asciicasts-endpoint [{:keys [db file-store exp-set]}] +(defn asciicasts-endpoint [{:keys [db file-store exp-set executor]}] (api {:exceptions {:handlers {:compojure.api.exception/default exception-handler}}} (context @@ -63,17 +64,19 @@ theme (assoc :theme theme) scale (assoc :scale (Integer/parseInt scale))) png-store-path (asciicast/png-store-path asciicast png-params)] - (when-not (exp-set/contains? exp-set png-store-path) - (with-tmp-dir [dir "asciinema-png-"] - (let [json-store-path (asciicast/json-store-path asciicast) - json-local-path (str dir "/asciicast.json") - png-local-path (str dir "/asciicast.png") - expires (-> png-ttl-days t/days t/from-now)] - (with-open [in (fstore/input-stream file-store json-store-path)] - (let [out (io/file json-local-path)] - (io/copy in out))) - (a2png json-local-path png-local-path png-params) - (fstore/put-file file-store (io/file png-local-path) png-store-path) - (exp-set/conj! exp-set png-store-path expires)))) - (fstore/serve-file file-store png-store-path {})) + (if (exp-set/contains? exp-set png-store-path) + (fstore/serve-file file-store png-store-path {}) + (executor/execute executor (fn [] + (with-tmp-dir [dir "asciinema-png-"] + (let [json-store-path (asciicast/json-store-path asciicast) + json-local-path (str dir "/asciicast.json") + png-local-path (str dir "/asciicast.png") + expires (-> png-ttl-days t/days t/from-now)] + (with-open [in (fstore/input-stream file-store json-store-path)] + (let [out (io/file json-local-path)] + (io/copy in out))) + (a2png json-local-path png-local-path png-params) + (fstore/put-file file-store (io/file png-local-path) png-store-path) + (exp-set/conj! exp-set png-store-path expires))) + (fstore/serve-file file-store png-store-path {}))))) (response/not-found)))))) From d343d89dea8fc1fd1efce992d595e01c0333bcae Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sun, 26 Feb 2017 11:43:47 +0100 Subject: [PATCH 23/46] cljr-clean-ns --- src/asciinema/endpoint/asciicasts.clj | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index eec30e4..ae97d96 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -1,15 +1,16 @@ (ns asciinema.endpoint.asciicasts (:require [asciinema.boundary [asciicast-database :as adb] + [executor :as executor] [expiring-set :as exp-set] [file-store :as fstore] - [executor :as executor] [user-database :as udb]] [asciinema.model.asciicast :as asciicast] [asciinema.util.io :refer [with-tmp-dir]] [clj-time.core :as t] - [clojure.java.io :as io] - [clojure.java.shell :as shell] + [clojure.java + [io :as io] + [shell :as shell]] [clojure.string :as str] [compojure.api.sweet :refer :all] [environ.core :refer [env]] From 3424505587a11b8083fb7465f1df1f6e6884ad93 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 28 Feb 2017 11:58:43 +0100 Subject: [PATCH 24/46] Use yada+bidi instead of compojure-api --- dev/resources/dev.edn | 6 +- project.clj | 12 +- resources/asciinema/system.edn | 46 +------ src/asciinema/boundary/file_store.clj | 2 +- src/asciinema/component/aleph.clj | 17 --- .../component/fixed_thread_executor.clj | 2 +- src/asciinema/component/local_file_store.clj | 9 +- src/asciinema/component/s3_file_store.clj | 9 +- src/asciinema/component/yada_listener.clj | 21 +++ src/asciinema/endpoint/asciicasts.clj | 126 ++++++++++-------- src/asciinema/model/asciicast.clj | 2 +- 11 files changed, 115 insertions(+), 137 deletions(-) delete mode 100644 src/asciinema/component/aleph.clj create mode 100644 src/asciinema/component/yada_listener.clj diff --git a/dev/resources/dev.edn b/dev/resources/dev.edn index 4733fa0..5f009e9 100644 --- a/dev/resources/dev.edn +++ b/dev/resources/dev.edn @@ -2,11 +2,7 @@ {:file-store #var asciinema.component.local-file-store/local-file-store :exp-set #var asciinema.component.mem-expiring-set/mem-expiring-set} :config - {:app - {:middleware - {:functions {:stacktrace #var ring.middleware.stacktrace/wrap-stacktrace} - :applied ^:replace [:not-found :webjars :ring-defaults :route-aliases :ring-logger :stacktrace]}} - :http + {:http {:port 4000} :db {:uri "jdbc:postgresql://localhost:15432/asciinema_development?user=vagrant"} diff --git a/project.clj b/project.clj index 0f74029..a4ba457 100644 --- a/project.clj +++ b/project.clj @@ -4,25 +4,19 @@ :min-lein-version "2.0.0" :dependencies [[org.clojure/clojure "1.8.0"] [com.stuartsierra/component "0.3.1"] - [metosin/ring-http-response "0.8.0"] [clj-time "0.13.0"] [duct "0.8.2"] - [compojure "1.5.1"] - [metosin/compojure-api "1.1.10"] + [yada "1.2.0"] + [aleph "0.4.1"] + [bidi "2.0.12"] [prismatic/schema "1.1.3"] [environ "1.1.0"] [ring "1.5.0"] - [ring/ring-defaults "0.2.1"] - [ring-jetty-component "0.3.1"] - [ring-webjars "0.1.1"] - [ring-logger-timbre "0.7.5"] [clj-bugsnag "0.2.9"] [clj-aws-s3 "0.3.10" :exclusions [joda-time com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.core/jackson-annotations]] - [aleph "0.4.1"] [pandect "0.6.1"] [com.taoensso/carmine "2.15.1"] [org.slf4j/slf4j-nop "1.7.21"] - [org.webjars/normalize.css "3.0.2"] [duct/hikaricp-component "0.1.0"] [org.postgresql/postgresql "9.4.1211"] [duct/ragtime-component "0.1.4"]] diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index eaebc66..aded10c 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -1,6 +1,5 @@ {:components - {:app #var duct.component.handler/handler-component - :http #var asciinema.component.aleph/aleph-server + {:http #var asciinema.component.yada-listener/yada-listener :db #var asciinema.component.db/hikaricp :ragtime #var duct.component.ragtime/ragtime :file-store #var asciinema.component.s3-file-store/s3-file-store @@ -9,50 +8,11 @@ :endpoints {:asciicasts #var asciinema.endpoint.asciicasts/asciicasts-endpoint} :dependencies - {:http [:app] - :app [:asciicasts] + {:http {:app :asciicasts} :ragtime [:db] :asciicasts [:db :file-store :exp-set :executor]} :config - {:app - {:middleware - {:functions - {:hide-errors #var duct.middleware.errors/wrap-hide-errors - :not-found #var duct.middleware.not-found/wrap-not-found - :ring-defaults #var ring.middleware.defaults/wrap-defaults - :route-aliases #var duct.middleware.route-aliases/wrap-route-aliases - :ring-logger #var ring.logger.timbre/wrap-with-logger - :bugsnag #var clj-bugsnag.ring/wrap-bugsnag - :webjars #var ring.middleware.webjars/wrap-webjars} - :applied - [:not-found :webjars :ring-defaults :route-aliases :ring-logger :bugsnag :hide-errors] - :arguments - {:not-found #resource "asciinema/errors/404.html" - :hide-errors #resource "asciinema/errors/500.html" - :bugsnag - {:api-key bugsnag-key - :environment env-name - :version git-sha - :project-ns "asciinema"} - :route-aliases {"/" "/index.html"} - :ring-defaults - {:params {:urlencoded true - :keywordize true - :multipart true - :nested true} - :cookies true - :session {:flash true - :cookie-attrs {:http-only true}} - :security {:anti-forgery true - :xss-protection {:enable? true, :mode :block} - :frame-options :sameorigin - :content-type-options :nosniff} - :static {:resources "asciinema/public"} - :responses {:not-modified-responses true - :absolute-redirects true - :content-types true - :default-charset "utf-8"}}}}} - :http + {:http {:port http-port} :db {:uri db-uri} diff --git a/src/asciinema/boundary/file_store.clj b/src/asciinema/boundary/file_store.clj index c9fad8c..da2d65d 100644 --- a/src/asciinema/boundary/file_store.clj +++ b/src/asciinema/boundary/file_store.clj @@ -5,4 +5,4 @@ (input-stream [this path]) (move-file [this old-path new-path]) (delete-file [this path]) - (serve-file [this path opts])) + (serve-file [this ctx path opts])) diff --git a/src/asciinema/component/aleph.clj b/src/asciinema/component/aleph.clj deleted file mode 100644 index a77bfbc..0000000 --- a/src/asciinema/component/aleph.clj +++ /dev/null @@ -1,17 +0,0 @@ -(ns asciinema.component.aleph - (:require [com.stuartsierra.component :as component] - [aleph.http :refer [start-server]])) - -(defrecord WebServer [port server app] - component/Lifecycle - (start [component] - (let [handler (:handler app) - server (start-server handler {:port port :join? false})] - (assoc component :server server))) - (stop [component] - (when server - (.close server) - component))) - -(defn aleph-server [{:keys [port app]}] - (map->WebServer {:port port :app app})) diff --git a/src/asciinema/component/fixed_thread_executor.clj b/src/asciinema/component/fixed_thread_executor.clj index fb7255a..7981882 100644 --- a/src/asciinema/component/fixed_thread_executor.clj +++ b/src/asciinema/component/fixed_thread_executor.clj @@ -21,7 +21,7 @@ (.execute executor f) result) (catch RejectedExecutionException _ - {:status 503 :headers {"Retry-After" "5"} :body "

503

"}))) + nil))) component/Lifecycle (start [{:keys [threads queue-length] :as component}] diff --git a/src/asciinema/component/local_file_store.clj b/src/asciinema/component/local_file_store.clj index 21823a0..bf82363 100644 --- a/src/asciinema/component/local_file_store.clj +++ b/src/asciinema/component/local_file_store.clj @@ -27,11 +27,12 @@ (let [path (str base-path path)] (io/delete-file path))) - (serve-file [this path {:keys [filename]}] - (let [resp (response/ok (file-store/input-stream this path))] + (serve-file [this ctx path {:keys [filename]}] + (let [path (str base-path path) + response (assoc (:response ctx) :body (io/file path))] (if filename - (response/header resp "Content-Disposition" (str "attachment; filename=" filename)) - resp)))) + (update response :headers assoc "content-disposition" (str "attachment; filename=" filename)) + response)))) (defn local-file-store [{:keys [path]}] (->LocalFileStore path)) diff --git a/src/asciinema/component/s3_file_store.clj b/src/asciinema/component/s3_file_store.clj index 4013f49..478892d 100644 --- a/src/asciinema/component/s3_file_store.clj +++ b/src/asciinema/component/s3_file_store.clj @@ -50,9 +50,12 @@ (let [path (str path-prefix path)] (s3/delete-object cred bucket path))) - (serve-file [this path opts] - (let [path (str path-prefix path)] - (response/found (generate-presigned-url cred bucket path opts))))) + (serve-file [this ctx path opts] + (let [path (str path-prefix path) + url (generate-presigned-url cred bucket path opts)] + (-> (:response ctx) + (assoc :status 302) + (update :headers assoc "location" url))))) (defn s3-file-store [{:keys [cred bucket path-prefix]}] (->S3FileStore cred bucket path-prefix)) diff --git a/src/asciinema/component/yada_listener.clj b/src/asciinema/component/yada_listener.clj new file mode 100644 index 0000000..d2c8810 --- /dev/null +++ b/src/asciinema/component/yada_listener.clj @@ -0,0 +1,21 @@ +(ns asciinema.component.yada-listener + (:require [com.stuartsierra.component :as component] + [yada.yada :as yada])) + +(defrecord YadaListener [port server app] + component/Lifecycle + (start [component] + (if server + component + (let [handler (:routes app) + server (yada/listener handler {:port port})] + (assoc component :server server)))) + (stop [component] + (if server + (do + ((:close server)) + (assoc component :server nil)) + component))) + +(defn yada-listener [{:keys [port app]}] + (map->YadaListener {:port port :app app})) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index ae97d96..acd43d7 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -11,16 +11,15 @@ [clojure.java [io :as io] [shell :as shell]] - [clojure.string :as str] - [compojure.api.sweet :refer :all] [environ.core :refer [env]] - [ring.util.http-response :as response] - [schema.core :as s])) + [schema.core :as s] + [yada.yada :as yada])) -(defn exception-handler [^Exception e data request] - (throw e)) +(def Theme (apply s/enum asciicast/themes)) -(defn a2png [in-url out-path {:keys [snapshot-at theme scale]}] +(def png-ttl-days 7) + +(defn- a2png [in-url out-path {:keys [snapshot-at theme scale]}] (let [a2png-bin (:a2png-bin env "a2png/a2png.sh") {:keys [exit] :as result} (shell/sh a2png-bin "-t" theme @@ -31,53 +30,74 @@ (when-not (zero? exit) (throw (ex-info "a2png error" result))))) -(def Num (s/if #(str/includes? % ".") - Double - s/Int)) +(defn- generate-png [file-store exp-set asciicast png-params png-store-path] + (with-tmp-dir [dir "asciinema-png-"] + (let [json-store-path (asciicast/json-store-path asciicast) + json-local-path (str dir "/asciicast.json") + png-local-path (str dir "/asciicast.png") + expires (-> png-ttl-days t/days t/from-now)] + (with-open [in (fstore/input-stream file-store json-store-path)] + (let [out (io/file json-local-path)] + (io/copy in out))) + (a2png json-local-path png-local-path png-params) + (fstore/put-file file-store (io/file png-local-path) png-store-path) + (exp-set/conj! exp-set png-store-path expires)))) -(def Theme (apply s/enum asciicast/themes)) +(defn- service-unavailable-response [ctx] + (-> (:response ctx) + (assoc :status 503) + (update :headers assoc "retry-after" "5"))) -(def png-ttl-days 7) +(defn- async-response [ctx executor f] + (or (executor/execute executor f) + (service-unavailable-response ctx))) + +(defn asciicast-json-resource [db file-store] + (yada/resource {:produces "application/json" + :parameters {:path {:token String} + :query {(s/optional-key :dl) s/Bool}} + :properties (fn [ctx] + (if-let [asciicast (adb/get-asciicast-by-token db (-> ctx :parameters :path :token))] + {:exists? true + ::asciicast asciicast} + {:exists? false})) + :response (fn [ctx] + (let [asciicast (-> ctx :properties ::asciicast) + dl (-> ctx :parameters :query :dl) + path (asciicast/json-store-path asciicast) + filename (str "asciicast-" (:id asciicast) ".json")] + (fstore/serve-file file-store ctx path (when dl {:filename filename}))))})) + +(defn asciicast-png-resource [db file-store exp-set executor] + (yada/resource {:produces "image/png" + :parameters {:path {:token String} + :query {(s/optional-key :time) s/Num + (s/optional-key :theme) Theme + (s/optional-key :scale) (s/enum "1" "2")}} + :properties (fn [ctx] + (if-let [asciicast (adb/get-asciicast-by-token db (-> ctx :parameters :path :token))] + (let [user (udb/get-user-by-id db (:user_id asciicast)) + {:keys [time theme scale]} (-> ctx :parameters :query) + png-params (cond-> (asciicast/png-params asciicast user) + time (assoc :snapshot-at time) + theme (assoc :theme theme) + scale (assoc :scale (Integer/parseInt scale)))] + {:exists? true + :version (asciicast/png-version asciicast png-params) + ::asciicast asciicast + ::png-params png-params}) + {:exists? false})) + :response (fn [ctx] + (let [asciicast (-> ctx :properties ::asciicast) + png-params (-> ctx :properties ::png-params) + png-store-path (asciicast/png-store-path asciicast png-params)] + (if (exp-set/contains? exp-set png-store-path) + (fstore/serve-file file-store ctx png-store-path {}) + (async-response ctx executor (fn [] + (generate-png file-store exp-set asciicast png-params png-store-path) + (fstore/serve-file file-store ctx png-store-path {}))))))})) (defn asciicasts-endpoint [{:keys [db file-store exp-set executor]}] - (api - {:exceptions {:handlers {:compojure.api.exception/default exception-handler}}} - (context - "/a" [] - (GET "/:token.json" [] - :path-params [token :- String] - :query-params [{dl :- s/Bool false}] - (if-let [asciicast (adb/get-asciicast-by-token db token)] - (let [path (asciicast/json-store-path asciicast) - filename (str "asciicast-" (:id asciicast) ".json")] - (fstore/serve-file file-store path (when dl {:filename filename}))) - (response/not-found))) - - (GET "/:token.png" [] - :path-params [token :- String] - :query-params [{time :- Num nil} - {theme :- Theme nil} - {scale :- (s/enum "1" "2") nil}] - (if-let [asciicast (adb/get-asciicast-by-token db token)] - (let [user (udb/get-user-by-id db (:user_id asciicast)) - png-params (cond-> (asciicast/png-params asciicast user) - time (assoc :snapshot-at time) - theme (assoc :theme theme) - scale (assoc :scale (Integer/parseInt scale))) - png-store-path (asciicast/png-store-path asciicast png-params)] - (if (exp-set/contains? exp-set png-store-path) - (fstore/serve-file file-store png-store-path {}) - (executor/execute executor (fn [] - (with-tmp-dir [dir "asciinema-png-"] - (let [json-store-path (asciicast/json-store-path asciicast) - json-local-path (str dir "/asciicast.json") - png-local-path (str dir "/asciicast.png") - expires (-> png-ttl-days t/days t/from-now)] - (with-open [in (fstore/input-stream file-store json-store-path)] - (let [out (io/file json-local-path)] - (io/copy in out))) - (a2png json-local-path png-local-path png-params) - (fstore/put-file file-store (io/file png-local-path) png-store-path) - (exp-set/conj! exp-set png-store-path expires))) - (fstore/serve-file file-store png-store-path {}))))) - (response/not-found)))))) + ["" [["/a/" [[[:token ".json"] (asciicast-json-resource db file-store)] + [[:token ".png"] (asciicast-png-resource db file-store exp-set executor)]]] + [true (yada/as-resource nil)]]]) diff --git a/src/asciinema/model/asciicast.clj b/src/asciinema/model/asciicast.clj index bd2bc19..8d28d11 100644 --- a/src/asciinema/model/asciicast.clj +++ b/src/asciinema/model/asciicast.clj @@ -25,7 +25,7 @@ :theme (theme-name asciicast user) :scale default-png-scale}) -(defn- png-version [asciicast params] +(defn png-version [asciicast params] (let [attrs (assoc params :id (:id asciicast))] (->> attrs (map (fn [[k v]] (str (name k) "=" v))) From bb4c602a4ce9e58f32aec4aaa111f0352dec6795 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 28 Feb 2017 17:56:29 +0100 Subject: [PATCH 25/46] Use wrapper for yada/resource --- src/asciinema/endpoint/asciicasts.clj | 85 ++++++++++++++------------- src/asciinema/yada.clj | 5 ++ 2 files changed, 49 insertions(+), 41 deletions(-) create mode 100644 src/asciinema/yada.clj diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index acd43d7..687e888 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -7,6 +7,7 @@ [user-database :as udb]] [asciinema.model.asciicast :as asciicast] [asciinema.util.io :refer [with-tmp-dir]] + [asciinema.yada :refer [resource]] [clj-time.core :as t] [clojure.java [io :as io] @@ -53,49 +54,51 @@ (service-unavailable-response ctx))) (defn asciicast-json-resource [db file-store] - (yada/resource {:produces "application/json" - :parameters {:path {:token String} - :query {(s/optional-key :dl) s/Bool}} - :properties (fn [ctx] - (if-let [asciicast (adb/get-asciicast-by-token db (-> ctx :parameters :path :token))] - {:exists? true - ::asciicast asciicast} - {:exists? false})) - :response (fn [ctx] - (let [asciicast (-> ctx :properties ::asciicast) - dl (-> ctx :parameters :query :dl) - path (asciicast/json-store-path asciicast) - filename (str "asciicast-" (:id asciicast) ".json")] - (fstore/serve-file file-store ctx path (when dl {:filename filename}))))})) + (resource + {:produces "application/json" + :parameters {:path {:token String} + :query {(s/optional-key :dl) s/Bool}} + :properties (fn [ctx] + (if-let [asciicast (adb/get-asciicast-by-token db (-> ctx :parameters :path :token))] + {:exists? true + ::asciicast asciicast} + {:exists? false})) + :response (fn [ctx] + (let [asciicast (-> ctx :properties ::asciicast) + dl (-> ctx :parameters :query :dl) + path (asciicast/json-store-path asciicast) + filename (str "asciicast-" (:id asciicast) ".json")] + (fstore/serve-file file-store ctx path (when dl {:filename filename}))))})) (defn asciicast-png-resource [db file-store exp-set executor] - (yada/resource {:produces "image/png" - :parameters {:path {:token String} - :query {(s/optional-key :time) s/Num - (s/optional-key :theme) Theme - (s/optional-key :scale) (s/enum "1" "2")}} - :properties (fn [ctx] - (if-let [asciicast (adb/get-asciicast-by-token db (-> ctx :parameters :path :token))] - (let [user (udb/get-user-by-id db (:user_id asciicast)) - {:keys [time theme scale]} (-> ctx :parameters :query) - png-params (cond-> (asciicast/png-params asciicast user) - time (assoc :snapshot-at time) - theme (assoc :theme theme) - scale (assoc :scale (Integer/parseInt scale)))] - {:exists? true - :version (asciicast/png-version asciicast png-params) - ::asciicast asciicast - ::png-params png-params}) - {:exists? false})) - :response (fn [ctx] - (let [asciicast (-> ctx :properties ::asciicast) - png-params (-> ctx :properties ::png-params) - png-store-path (asciicast/png-store-path asciicast png-params)] - (if (exp-set/contains? exp-set png-store-path) - (fstore/serve-file file-store ctx png-store-path {}) - (async-response ctx executor (fn [] - (generate-png file-store exp-set asciicast png-params png-store-path) - (fstore/serve-file file-store ctx png-store-path {}))))))})) + (resource + {:produces "image/png" + :parameters {:path {:token String} + :query {(s/optional-key :time) s/Num + (s/optional-key :theme) Theme + (s/optional-key :scale) (s/enum "1" "2")}} + :properties (fn [ctx] + (if-let [asciicast (adb/get-asciicast-by-token db (-> ctx :parameters :path :token))] + (let [user (udb/get-user-by-id db (:user_id asciicast)) + {:keys [time theme scale]} (-> ctx :parameters :query) + png-params (cond-> (asciicast/png-params asciicast user) + time (assoc :snapshot-at time) + theme (assoc :theme theme) + scale (assoc :scale (Integer/parseInt scale)))] + {:exists? true + :version (asciicast/png-version asciicast png-params) + ::asciicast asciicast + ::png-params png-params}) + {:exists? false})) + :response (fn [ctx] + (let [asciicast (-> ctx :properties ::asciicast) + png-params (-> ctx :properties ::png-params) + png-store-path (asciicast/png-store-path asciicast png-params)] + (if (exp-set/contains? exp-set png-store-path) + (fstore/serve-file file-store ctx png-store-path {}) + (async-response ctx executor (fn [] + (generate-png file-store exp-set asciicast png-params png-store-path) + (fstore/serve-file file-store ctx png-store-path {}))))))})) (defn asciicasts-endpoint [{:keys [db file-store exp-set executor]}] ["" [["/a/" [[[:token ".json"] (asciicast-json-resource db file-store)] diff --git a/src/asciinema/yada.clj b/src/asciinema/yada.clj new file mode 100644 index 0000000..e440063 --- /dev/null +++ b/src/asciinema/yada.clj @@ -0,0 +1,5 @@ +(ns asciinema.yada + (:require [yada.yada :as yada])) + +(defn resource [model] + (yada/resource model)) From 1e08b39e4990cf4acb215a14752b25c6a78c8a80 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 28 Feb 2017 18:32:37 +0100 Subject: [PATCH 26/46] Default 404 page --- src/asciinema/yada.clj | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/asciinema/yada.clj b/src/asciinema/yada.clj index e440063..e15788e 100644 --- a/src/asciinema/yada.clj +++ b/src/asciinema/yada.clj @@ -1,5 +1,15 @@ (ns asciinema.yada - (:require [yada.yada :as yada])) + (:require [clojure.java.io :as io] + [yada.yada :as yada])) + +(defn not-found-response [ctx] + (case (yada/content-type ctx) + "text/html" (io/input-stream (io/resource "asciinema/errors/404.html")) + "Not found")) (defn resource [model] - (yada/resource model)) + (-> model + (update-in [:responses 404] #(or % + {:produces #{"text/html" "text/plain"} + :response not-found-response})) + yada/resource)) From 4fe17fbc82a17a187dcd9a6d4fad1b925f9163b1 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 28 Feb 2017 18:35:32 +0100 Subject: [PATCH 27/46] Serve 404 page when none route matches --- src/asciinema/endpoint/asciicasts.clj | 4 ++-- src/asciinema/yada.clj | 18 +++++++++++------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index 687e888..a4517ae 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -7,7 +7,7 @@ [user-database :as udb]] [asciinema.model.asciicast :as asciicast] [asciinema.util.io :refer [with-tmp-dir]] - [asciinema.yada :refer [resource]] + [asciinema.yada :refer [resource not-found-model]] [clj-time.core :as t] [clojure.java [io :as io] @@ -103,4 +103,4 @@ (defn asciicasts-endpoint [{:keys [db file-store exp-set executor]}] ["" [["/a/" [[[:token ".json"] (asciicast-json-resource db file-store)] [[:token ".png"] (asciicast-png-resource db file-store exp-set executor)]]] - [true (yada/as-resource nil)]]]) + [true (yada/resource not-found-model)]]]) diff --git a/src/asciinema/yada.clj b/src/asciinema/yada.clj index e15788e..e2dafcc 100644 --- a/src/asciinema/yada.clj +++ b/src/asciinema/yada.clj @@ -2,14 +2,18 @@ (:require [clojure.java.io :as io] [yada.yada :as yada])) -(defn not-found-response [ctx] - (case (yada/content-type ctx) - "text/html" (io/input-stream (io/resource "asciinema/errors/404.html")) - "Not found")) +(def not-found-model + {:produces + #{"text/html" "text/plain"} + :response + (fn [ctx] + (assoc (:response ctx) + :status 404 + :body (case (yada/content-type ctx) + "text/html" (io/input-stream (io/resource "asciinema/errors/404.html")) + "Not found")))}) (defn resource [model] (-> model - (update-in [:responses 404] #(or % - {:produces #{"text/html" "text/plain"} - :response not-found-response})) + (update-in [:responses 404] #(or % not-found-model)) yada/resource)) From 387d47757fa326a6c05026620b92207940b64899 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 1 Mar 2017 12:14:29 +0100 Subject: [PATCH 28/46] No need to return `:exists? true` when resource exists --- src/asciinema/endpoint/asciicasts.clj | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index a4517ae..0a038b5 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -60,8 +60,7 @@ :query {(s/optional-key :dl) s/Bool}} :properties (fn [ctx] (if-let [asciicast (adb/get-asciicast-by-token db (-> ctx :parameters :path :token))] - {:exists? true - ::asciicast asciicast} + {::asciicast asciicast} {:exists? false})) :response (fn [ctx] (let [asciicast (-> ctx :properties ::asciicast) @@ -85,8 +84,7 @@ time (assoc :snapshot-at time) theme (assoc :theme theme) scale (assoc :scale (Integer/parseInt scale)))] - {:exists? true - :version (asciicast/png-version asciicast png-params) + {:version (asciicast/png-version asciicast png-params) ::asciicast asciicast ::png-params png-params}) {:exists? false})) From 0b6ef2492aac3080605dd875013fc919effd7c22 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 1 Mar 2017 12:32:11 +0100 Subject: [PATCH 29/46] Fix clj-aws-s3 deps --- project.clj | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/project.clj b/project.clj index a4ba457..5a84160 100644 --- a/project.clj +++ b/project.clj @@ -13,7 +13,7 @@ [environ "1.1.0"] [ring "1.5.0"] [clj-bugsnag "0.2.9"] - [clj-aws-s3 "0.3.10" :exclusions [joda-time com.fasterxml.jackson.core/jackson-core com.fasterxml.jackson.core/jackson-annotations]] + [clj-aws-s3 "0.3.10" :exclusions [joda-time]] [pandect "0.6.1"] [com.taoensso/carmine "2.15.1"] [org.slf4j/slf4j-nop "1.7.21"] From 137ee108c5be73176a1ffb3eaf3c15b906322cd0 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 1 Mar 2017 16:20:26 +0100 Subject: [PATCH 30/46] Refactor png generation --- resources/asciinema/system.edn | 5 +- src/asciinema/boundary/png_generator.clj | 4 + src/asciinema/component/a2png.clj | 30 +++++++ src/asciinema/endpoint/asciicasts.clj | 110 ++++++++++------------- src/asciinema/util/io.clj | 11 ++- 5 files changed, 95 insertions(+), 65 deletions(-) create mode 100644 src/asciinema/boundary/png_generator.clj create mode 100644 src/asciinema/component/a2png.clj diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index aded10c..3afef71 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -4,13 +4,14 @@ :ragtime #var duct.component.ragtime/ragtime :file-store #var asciinema.component.s3-file-store/s3-file-store :exp-set #var asciinema.component.redis-client/redis-client + :png-gen #var asciinema.component.a2png/a2png :executor #var asciinema.component.fixed-thread-executor/fixed-thread-executor} :endpoints {:asciicasts #var asciinema.endpoint.asciicasts/asciicasts-endpoint} :dependencies {:http {:app :asciicasts} :ragtime [:db] - :asciicasts [:db :file-store :exp-set :executor]} + :asciicasts [:db :file-store :exp-set :executor :png-gen]} :config {:http {:port http-port} @@ -26,6 +27,8 @@ :exp-set {:host redis-host :port redis-port} + :png-gen + {:bin-path "a2png/a2png.sh"} :executor {:threads 1 :queue-length 1}}} diff --git a/src/asciinema/boundary/png_generator.clj b/src/asciinema/boundary/png_generator.clj new file mode 100644 index 0000000..c975c29 --- /dev/null +++ b/src/asciinema/boundary/png_generator.clj @@ -0,0 +1,4 @@ +(ns asciinema.boundary.png-generator) + +(defprotocol PngGenerator + (generate [this json-is png-params])) diff --git a/src/asciinema/component/a2png.clj b/src/asciinema/component/a2png.clj new file mode 100644 index 0000000..b5cff40 --- /dev/null +++ b/src/asciinema/component/a2png.clj @@ -0,0 +1,30 @@ +(ns asciinema.component.a2png + (:require [asciinema.boundary.png-generator :as png-generator] + [asciinema.util.io :refer [cleanup-input-stream create-tmp-dir]] + [clojure.java + [io :as io] + [shell :as shell]])) + +(defn- exec-a2png [bin-path in-url out-path {:keys [snapshot-at theme scale]}] + (let [{:keys [exit] :as result} (shell/sh bin-path + "-t" theme + "-s" (str scale) + in-url + out-path + (str snapshot-at))] + (when-not (zero? exit) + (throw (ex-info "a2png error" result))))) + +(defrecord A2png [bin-path] + png-generator/PngGenerator + (generate [this json-is png-params] + (let [dir (create-tmp-dir "a2png-") + cleanup #(shell/sh "rm" "-rf" (.getPath dir)) + json-local-path (str dir "/asciicast.json") + png-local-path (str dir "/asciicast.png")] + (io/copy json-is (io/file json-local-path)) + (exec-a2png bin-path json-local-path png-local-path png-params) + (cleanup-input-stream (io/input-stream png-local-path) cleanup)))) + +(defn a2png [{:keys [bin-path]}] + (->A2png bin-path)) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index 0a038b5..e7ecd38 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -4,46 +4,16 @@ [executor :as executor] [expiring-set :as exp-set] [file-store :as fstore] + [png-generator :as png] [user-database :as udb]] [asciinema.model.asciicast :as asciicast] - [asciinema.util.io :refer [with-tmp-dir]] - [asciinema.yada :refer [resource not-found-model]] + [asciinema.yada :refer [not-found-model resource]] [clj-time.core :as t] - [clojure.java - [io :as io] - [shell :as shell]] - [environ.core :refer [env]] [schema.core :as s] [yada.yada :as yada])) (def Theme (apply s/enum asciicast/themes)) -(def png-ttl-days 7) - -(defn- a2png [in-url out-path {:keys [snapshot-at theme scale]}] - (let [a2png-bin (:a2png-bin env "a2png/a2png.sh") - {:keys [exit] :as result} (shell/sh a2png-bin - "-t" theme - "-s" (str scale) - in-url - out-path - (str snapshot-at))] - (when-not (zero? exit) - (throw (ex-info "a2png error" result))))) - -(defn- generate-png [file-store exp-set asciicast png-params png-store-path] - (with-tmp-dir [dir "asciinema-png-"] - (let [json-store-path (asciicast/json-store-path asciicast) - json-local-path (str dir "/asciicast.json") - png-local-path (str dir "/asciicast.png") - expires (-> png-ttl-days t/days t/from-now)] - (with-open [in (fstore/input-stream file-store json-store-path)] - (let [out (io/file json-local-path)] - (io/copy in out))) - (a2png json-local-path png-local-path png-params) - (fstore/put-file file-store (io/file png-local-path) png-store-path) - (exp-set/conj! exp-set png-store-path expires)))) - (defn- service-unavailable-response [ctx] (-> (:response ctx) (assoc :status 503) @@ -69,36 +39,52 @@ filename (str "asciicast-" (:id asciicast) ".json")] (fstore/serve-file file-store ctx path (when dl {:filename filename}))))})) -(defn asciicast-png-resource [db file-store exp-set executor] - (resource - {:produces "image/png" - :parameters {:path {:token String} - :query {(s/optional-key :time) s/Num - (s/optional-key :theme) Theme - (s/optional-key :scale) (s/enum "1" "2")}} - :properties (fn [ctx] - (if-let [asciicast (adb/get-asciicast-by-token db (-> ctx :parameters :path :token))] - (let [user (udb/get-user-by-id db (:user_id asciicast)) - {:keys [time theme scale]} (-> ctx :parameters :query) - png-params (cond-> (asciicast/png-params asciicast user) - time (assoc :snapshot-at time) - theme (assoc :theme theme) - scale (assoc :scale (Integer/parseInt scale)))] - {:version (asciicast/png-version asciicast png-params) - ::asciicast asciicast - ::png-params png-params}) - {:exists? false})) - :response (fn [ctx] - (let [asciicast (-> ctx :properties ::asciicast) - png-params (-> ctx :properties ::png-params) - png-store-path (asciicast/png-store-path asciicast png-params)] - (if (exp-set/contains? exp-set png-store-path) - (fstore/serve-file file-store ctx png-store-path {}) - (async-response ctx executor (fn [] - (generate-png file-store exp-set asciicast png-params png-store-path) - (fstore/serve-file file-store ctx png-store-path {}))))))})) +(def png-ttl-days 7) -(defn asciicasts-endpoint [{:keys [db file-store exp-set executor]}] +(defn asciicast-png-resource [db file-store exp-set executor png-gen] + (resource + {:produces + "image/png" + + :parameters + {:path {:token String} + :query {(s/optional-key :time) s/Num + (s/optional-key :theme) Theme + (s/optional-key :scale) (s/enum "1" "2")}} + + :properties + (fn [ctx] + (if-let [asciicast (adb/get-asciicast-by-token db (-> ctx :parameters :path :token))] + (let [user (udb/get-user-by-id db (:user_id asciicast)) + {:keys [time theme scale]} (-> ctx :parameters :query) + png-params (cond-> (asciicast/png-params asciicast user) + time (assoc :snapshot-at time) + theme (assoc :theme theme) + scale (assoc :scale (Integer/parseInt scale)))] + {:version (asciicast/png-version asciicast png-params) + ::asciicast asciicast + ::png-params png-params}) + {:exists? false})) + + :response + (fn [ctx] + (let [asciicast (-> ctx :properties ::asciicast) + png-params (-> ctx :properties ::png-params) + png-store-path (asciicast/png-store-path asciicast png-params) + expires (-> png-ttl-days t/days t/from-now)] + (if (exp-set/contains? exp-set png-store-path) + (fstore/serve-file file-store ctx png-store-path {}) + (async-response ctx + executor + (fn [] + (let [json-store-path (asciicast/json-store-path asciicast)] + (with-open [json-is (fstore/input-stream file-store json-store-path) + png-is (png/generate png-gen json-is png-params)] + (fstore/put-file file-store png-is png-store-path))) + (exp-set/conj! exp-set png-store-path expires) + (fstore/serve-file file-store ctx png-store-path {}))))))})) + +(defn asciicasts-endpoint [{:keys [db file-store exp-set executor png-gen]}] ["" [["/a/" [[[:token ".json"] (asciicast-json-resource db file-store)] - [[:token ".png"] (asciicast-png-resource db file-store exp-set executor)]]] + [[:token ".png"] (asciicast-png-resource db file-store exp-set executor png-gen)]]] [true (yada/resource not-found-model)]]]) diff --git a/src/asciinema/util/io.clj b/src/asciinema/util/io.clj index cdc4676..6c2d1ad 100644 --- a/src/asciinema/util/io.clj +++ b/src/asciinema/util/io.clj @@ -1,7 +1,8 @@ (ns asciinema.util.io (:require [clojure.java.shell :as shell]) - (:import java.nio.file.Files - java.nio.file.attribute.FileAttribute)) + (:import java.io.FilterInputStream + java.nio.file.attribute.FileAttribute + java.nio.file.Files)) (defn create-tmp-dir [prefix] (let [dir (Files/createTempDirectory prefix (into-array FileAttribute []))] @@ -13,3 +14,9 @@ ~@body (finally (shell/sh "rm" "-rf" (.getPath ~sym)))))) + +(defn cleanup-input-stream [is cleanup] + (proxy [FilterInputStream] [is] + (close [] + (proxy-super close) + (cleanup)))) From d33e03ff2e2de3fc29a435abc4246018c3a9b540 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 1 Mar 2017 16:22:42 +0100 Subject: [PATCH 31/46] Use latest cheshire version (yada chokes on 5.6.3) --- project.clj | 1 + 1 file changed, 1 insertion(+) diff --git a/project.clj b/project.clj index 5a84160..a03d162 100644 --- a/project.clj +++ b/project.clj @@ -14,6 +14,7 @@ [ring "1.5.0"] [clj-bugsnag "0.2.9"] [clj-aws-s3 "0.3.10" :exclusions [joda-time]] + [cheshire "5.7.0"] [pandect "0.6.1"] [com.taoensso/carmine "2.15.1"] [org.slf4j/slf4j-nop "1.7.21"] From b39630d9a7133c58ba493cbd568cccba968a36bd Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Thu, 2 Mar 2017 11:53:43 +0100 Subject: [PATCH 32/46] Log errors with clojure.tools.logging/error --- src/asciinema/yada.clj | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/asciinema/yada.clj b/src/asciinema/yada.clj index e2dafcc..58baf6d 100644 --- a/src/asciinema/yada.clj +++ b/src/asciinema/yada.clj @@ -1,5 +1,6 @@ (ns asciinema.yada (:require [clojure.java.io :as io] + [clojure.tools.logging :as log] [yada.yada :as yada])) (def not-found-model @@ -13,7 +14,13 @@ "text/html" (io/input-stream (io/resource "asciinema/errors/404.html")) "Not found")))}) +(defn logger [ctx] + (when-let [error (:error ctx)] + (when (= (-> ctx :response :status) 500) + (log/error error)))) + (defn resource [model] (-> model + (assoc :logger logger) (update-in [:responses 404] #(or % not-found-model)) yada/resource)) From 964947009baf8a00b5e9d4b96df775008d50017c Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Thu, 2 Mar 2017 12:08:27 +0100 Subject: [PATCH 33/46] Use latest bidi --- project.clj | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/project.clj b/project.clj index a03d162..bf517c0 100644 --- a/project.clj +++ b/project.clj @@ -8,7 +8,7 @@ [duct "0.8.2"] [yada "1.2.0"] [aleph "0.4.1"] - [bidi "2.0.12"] + [bidi "2.0.16"] [prismatic/schema "1.1.3"] [environ "1.1.0"] [ring "1.5.0"] From 71479150f30892b8fb202b23670be42a25769eef Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Thu, 2 Mar 2017 12:23:36 +0100 Subject: [PATCH 34/46] Use timbre directly --- project.clj | 1 + src/asciinema/yada.clj | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/project.clj b/project.clj index bf517c0..62eac59 100644 --- a/project.clj +++ b/project.clj @@ -16,6 +16,7 @@ [clj-aws-s3 "0.3.10" :exclusions [joda-time]] [cheshire "5.7.0"] [pandect "0.6.1"] + [com.taoensso/timbre "4.8.0"] [com.taoensso/carmine "2.15.1"] [org.slf4j/slf4j-nop "1.7.21"] [duct/hikaricp-component "0.1.0"] diff --git a/src/asciinema/yada.clj b/src/asciinema/yada.clj index 58baf6d..afea114 100644 --- a/src/asciinema/yada.clj +++ b/src/asciinema/yada.clj @@ -1,6 +1,6 @@ (ns asciinema.yada (:require [clojure.java.io :as io] - [clojure.tools.logging :as log] + [taoensso.timbre :as log] [yada.yada :as yada])) (def not-found-model From 6769b08818c7ce49be700cadab45c44fe554ee90 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Thu, 2 Mar 2017 12:44:16 +0100 Subject: [PATCH 35/46] Log errors except 404 --- src/asciinema/yada.clj | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/asciinema/yada.clj b/src/asciinema/yada.clj index afea114..b310196 100644 --- a/src/asciinema/yada.clj +++ b/src/asciinema/yada.clj @@ -16,7 +16,7 @@ (defn logger [ctx] (when-let [error (:error ctx)] - (when (= (-> ctx :response :status) 500) + (when (not= (-> ctx :response :status) 404) (log/error error)))) (defn resource [model] From 800f466fa9ce551e3159d7b5e07e93c95ba3addd Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Thu, 2 Mar 2017 12:45:15 +0100 Subject: [PATCH 36/46] Custom responses for error statuses --- src/asciinema/yada.clj | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/src/asciinema/yada.clj b/src/asciinema/yada.clj index b310196..573a0ee 100644 --- a/src/asciinema/yada.clj +++ b/src/asciinema/yada.clj @@ -1,6 +1,7 @@ (ns asciinema.yada (:require [clojure.java.io :as io] [taoensso.timbre :as log] + [yada.status :as status] [yada.yada :as yada])) (def not-found-model @@ -14,13 +15,23 @@ "text/html" (io/input-stream (io/resource "asciinema/errors/404.html")) "Not found")))}) +(defn error-response [ctx] + (let [status (-> ctx :response :status) + status-name (get-in status/status [status :name])] + (case (yada/content-type ctx) + "text/html" (str "

" status-name "

") + status-name))) + (defn logger [ctx] (when-let [error (:error ctx)] (when (not= (-> ctx :response :status) 404) (log/error error)))) (defn resource [model] - (-> model - (assoc :logger logger) - (update-in [:responses 404] #(or % not-found-model)) - yada/resource)) + (let [error-statuses (set (concat (range 400 404) (range 405 600) ))] + (-> model + (assoc :logger logger) + (update-in [:responses 404] #(or % not-found-model)) + (update-in [:responses error-statuses] #(or % {:produces #{"text/html" "text/plain"} + :response error-response})) + yada/resource))) From 767e33145f99e3acf67e56c81f295d8cd8272d7d Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sat, 4 Mar 2017 13:07:26 +0100 Subject: [PATCH 37/46] Integrate yada with Bugsnag notifier --- src/asciinema/main.clj | 45 +++++++++++++++++++++++++++++------------- src/asciinema/yada.clj | 19 +++++++++++++----- 2 files changed, 45 insertions(+), 19 deletions(-) diff --git a/src/asciinema/main.clj b/src/asciinema/main.clj index 0a84342..41d70d4 100644 --- a/src/asciinema/main.clj +++ b/src/asciinema/main.clj @@ -1,24 +1,41 @@ (ns asciinema.main (:gen-class) - (:require [com.stuartsierra.component :as component] + (:require [asciinema.yada :as y] + [clj-bugsnag.core :as bugsnag] + [com.stuartsierra.component :as component] [duct.util.runtime :refer [add-shutdown-hook]] [duct.util.system :refer [load-system]] [environ.core :refer [env]] [clojure.java.io :as io])) +(defn- request-context [req] + (str (-> req (get :request-method :unknown) name .toUpperCase) + " " + (:uri req))) + +(defn- create-exception-notifier [] + (when-let [key (:bugsnag-key env)] + (let [environment (:env-name env "production") + version (:git-sha env)] + (fn [ex req] + (bugsnag/notify ex {:api-key key + :environment environment + :project-ns "asciinema" + :version version + :context (request-context req) + :meta {:request (dissoc req :body)}}))))) + (defn -main [& args] - (let [bindings {'http-port (Integer/parseInt (:port env "3000")) - 'db-uri (:database-url env) - 'env-name (:env-name env "production") - 'git-sha (:git-sha env) - 'bugsnag-key (:bugsnag-key env) - 's3-bucket (:s3-bucket env) - 's3-access-key (:s3-access-key env) - 's3-secret-key (:s3-secret-key env) - 'redis-host (:redis-host env "localhost") - 'redis-port (Integer/parseInt (:redis-port env "6379"))} - system (->> (load-system [(io/resource "asciinema/system.edn")] bindings) + (binding [y/*exception-notifier* (create-exception-notifier)] + (let [bindings {'http-port (Integer/parseInt (:port env "3000")) + 'db-uri (:database-url env) + 's3-bucket (:s3-bucket env) + 's3-access-key (:s3-access-key env) + 's3-secret-key (:s3-secret-key env) + 'redis-host (:redis-host env "localhost") + 'redis-port (Integer/parseInt (:redis-port env "6379"))} + system (->> (load-system [(io/resource "asciinema/system.edn")] bindings) (component/start))] - (add-shutdown-hook ::stop-system #(component/stop system)) - (println "Started HTTP server on port" (-> system :http :port))) + (add-shutdown-hook ::stop-system #(component/stop system)) + (println "Started HTTP server on port" (-> system :http :port)))) @(promise)) diff --git a/src/asciinema/yada.clj b/src/asciinema/yada.clj index 573a0ee..06d7057 100644 --- a/src/asciinema/yada.clj +++ b/src/asciinema/yada.clj @@ -4,6 +4,8 @@ [yada.status :as status] [yada.yada :as yada])) +(def ^:dynamic *exception-notifier* nil) + (def not-found-model {:produces #{"text/html" "text/plain"} @@ -22,15 +24,22 @@ "text/html" (str "

" status-name "

") status-name))) -(defn logger [ctx] - (when-let [error (:error ctx)] - (when (not= (-> ctx :response :status) 404) - (log/error error)))) +(defn create-logger [] + (let [notifier *exception-notifier*] + (fn [ctx] + (when-let [error (:error ctx)] + (let [status (-> ctx :response :status)] + (when (not= status 404) + (log/error error)) + (when (and (= status 500) notifier) + (let [ex (or (-> error ex-data :error) error)] + (notifier ex (:request ctx)))))) + ctx))) (defn resource [model] (let [error-statuses (set (concat (range 400 404) (range 405 600) ))] (-> model - (assoc :logger logger) + (assoc :logger (create-logger)) (update-in [:responses 404] #(or % not-found-model)) (update-in [:responses error-statuses] #(or % {:produces #{"text/html" "text/plain"} :response error-response})) From bb0a8b7e299de78f8b6ac99644c9e66409d305fe Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sat, 4 Mar 2017 13:10:19 +0100 Subject: [PATCH 38/46] Use 2 threads for generating PNG --- resources/asciinema/system.edn | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index 3afef71..ed3d3d8 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -30,5 +30,5 @@ :png-gen {:bin-path "a2png/a2png.sh"} :executor - {:threads 1 - :queue-length 1}}} + {:threads 2 + :queue-length 16}}} From 3d1314134963b188ff599431ebf54356f134b591 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sat, 4 Mar 2017 14:07:43 +0100 Subject: [PATCH 39/46] Allow setting a2png path via A2PNG_BIN_PATH --- resources/asciinema/system.edn | 2 +- src/asciinema/main.clj | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/resources/asciinema/system.edn b/resources/asciinema/system.edn index ed3d3d8..aa5b2c8 100644 --- a/resources/asciinema/system.edn +++ b/resources/asciinema/system.edn @@ -28,7 +28,7 @@ {:host redis-host :port redis-port} :png-gen - {:bin-path "a2png/a2png.sh"} + {:bin-path a2png-bin-path} :executor {:threads 2 :queue-length 16}}} diff --git a/src/asciinema/main.clj b/src/asciinema/main.clj index 41d70d4..7eecb3e 100644 --- a/src/asciinema/main.clj +++ b/src/asciinema/main.clj @@ -33,7 +33,8 @@ 's3-access-key (:s3-access-key env) 's3-secret-key (:s3-secret-key env) 'redis-host (:redis-host env "localhost") - 'redis-port (Integer/parseInt (:redis-port env "6379"))} + 'redis-port (Integer/parseInt (:redis-port env "6379")) + 'a2png-bin-path (:a2png-bin-path env "a2png/a2png.sh")} system (->> (load-system [(io/resource "asciinema/system.edn")] bindings) (component/start))] (add-shutdown-hook ::stop-system #(component/stop system)) From 60b8c2cc047e09c2c1939fffed9a4a3d20d3c7c6 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Sat, 4 Mar 2017 18:25:49 +0100 Subject: [PATCH 40/46] Set content-type for files uploaded to S3 --- src/asciinema/component/s3_file_store.clj | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/asciinema/component/s3_file_store.clj b/src/asciinema/component/s3_file_store.clj index 478892d..3443dd0 100644 --- a/src/asciinema/component/s3_file_store.clj +++ b/src/asciinema/component/s3_file_store.clj @@ -4,7 +4,8 @@ [clj-time [coerce :as timec] [core :as time]] - [ring.util.http-response :as response]) + [ring.util.http-response :as response] + [ring.util.mime-type :as mime-type]) (:import com.amazonaws.auth.BasicAWSCredentials com.amazonaws.services.s3.AmazonS3Client [com.amazonaws.services.s3.model GeneratePresignedUrlRequest ResponseHeaderOverrides])) @@ -33,8 +34,10 @@ (file-store/put-file this file path nil)) (put-file [this file path size] - (let [path (str path-prefix path)] - (s3/put-object cred bucket path file {:content-length size}))) + (let [path (str path-prefix path) + content-type (mime-type/ext-mime-type path)] + (s3/put-object cred bucket path file {:content-length size + :content-type content-type}))) (input-stream [this path] (let [path (str path-prefix path)] From 290cd74126dceaf41bb37d6ee2457f6da144c863 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Fri, 10 Mar 2017 23:04:50 +0100 Subject: [PATCH 41/46] Set :png-gen/:bin-path for dev env --- dev/resources/dev.edn | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/dev/resources/dev.edn b/dev/resources/dev.edn index 5f009e9..cf8c240 100644 --- a/dev/resources/dev.edn +++ b/dev/resources/dev.edn @@ -7,4 +7,6 @@ :db {:uri "jdbc:postgresql://localhost:15432/asciinema_development?user=vagrant"} :file-store - {:path "uploads/"}}} + {:path "uploads/"} + :png-gen + {:bin-path "a2png/a2png.sh"}}} From bc6f1ed3c728429299ca738fe0bf3da174df55a8 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Fri, 10 Mar 2017 23:08:31 +0100 Subject: [PATCH 42/46] Rename resource fns --- src/asciinema/endpoint/asciicasts.clj | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/asciinema/endpoint/asciicasts.clj b/src/asciinema/endpoint/asciicasts.clj index e7ecd38..572643e 100644 --- a/src/asciinema/endpoint/asciicasts.clj +++ b/src/asciinema/endpoint/asciicasts.clj @@ -23,7 +23,7 @@ (or (executor/execute executor f) (service-unavailable-response ctx))) -(defn asciicast-json-resource [db file-store] +(defn asciicast-file-resource [db file-store] (resource {:produces "application/json" :parameters {:path {:token String} @@ -41,7 +41,7 @@ (def png-ttl-days 7) -(defn asciicast-png-resource [db file-store exp-set executor png-gen] +(defn asciicast-image-resource [db file-store exp-set executor png-gen] (resource {:produces "image/png" @@ -85,6 +85,6 @@ (fstore/serve-file file-store ctx png-store-path {}))))))})) (defn asciicasts-endpoint [{:keys [db file-store exp-set executor png-gen]}] - ["" [["/a/" [[[:token ".json"] (asciicast-json-resource db file-store)] - [[:token ".png"] (asciicast-png-resource db file-store exp-set executor png-gen)]]] + ["" [["/a/" [[[:token ".json"] (asciicast-file-resource db file-store)] + [[:token ".png"] (asciicast-image-resource db file-store exp-set executor png-gen)]]] [true (yada/resource not-found-model)]]]) From 8146b66b70fbb8c6f6a79b99f1048281f1d5fd66 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 11 Apr 2017 12:56:04 +0200 Subject: [PATCH 43/46] Fix arguments for a2png --- src/asciinema/component/a2png.clj | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/asciinema/component/a2png.clj b/src/asciinema/component/a2png.clj index b5cff40..469b668 100644 --- a/src/asciinema/component/a2png.clj +++ b/src/asciinema/component/a2png.clj @@ -7,11 +7,11 @@ (defn- exec-a2png [bin-path in-url out-path {:keys [snapshot-at theme scale]}] (let [{:keys [exit] :as result} (shell/sh bin-path - "-t" theme - "-s" (str scale) in-url out-path - (str snapshot-at))] + (str snapshot-at) + theme + (str scale))] (when-not (zero? exit) (throw (ex-info "a2png error" result))))) From 6d15e618607a6e08a22e2ce1b1988029e905c82e Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Tue, 11 Apr 2017 14:43:10 +0200 Subject: [PATCH 44/46] Kill a2png after 30 sec --- project.clj | 3 ++- src/asciinema/component/a2png.clj | 14 +++++--------- 2 files changed, 7 insertions(+), 10 deletions(-) diff --git a/project.clj b/project.clj index 62eac59..f90bd48 100644 --- a/project.clj +++ b/project.clj @@ -21,7 +21,8 @@ [org.slf4j/slf4j-nop "1.7.21"] [duct/hikaricp-component "0.1.0"] [org.postgresql/postgresql "9.4.1211"] - [duct/ragtime-component "0.1.4"]] + [duct/ragtime-component "0.1.4"] + [me.raynes/conch "0.8.0"]] :plugins [[lein-environ "1.0.3"]] :main ^:skip-aot asciinema.main :target-path "target/%s/" diff --git a/src/asciinema/component/a2png.clj b/src/asciinema/component/a2png.clj index 469b668..65acaf2 100644 --- a/src/asciinema/component/a2png.clj +++ b/src/asciinema/component/a2png.clj @@ -1,19 +1,15 @@ (ns asciinema.component.a2png (:require [asciinema.boundary.png-generator :as png-generator] [asciinema.util.io :refer [cleanup-input-stream create-tmp-dir]] + [clojure.java.io :as io] [clojure.java [io :as io] - [shell :as shell]])) + [shell :as shell]] + [me.raynes.conch :as conch])) (defn- exec-a2png [bin-path in-url out-path {:keys [snapshot-at theme scale]}] - (let [{:keys [exit] :as result} (shell/sh bin-path - in-url - out-path - (str snapshot-at) - theme - (str scale))] - (when-not (zero? exit) - (throw (ex-info "a2png error" result))))) + (conch/let-programs [a2png bin-path] + (a2png in-url out-path (str snapshot-at) theme (str scale) {:timeout 30000}))) (defrecord A2png [bin-path] png-generator/PngGenerator From 7345df6260dbcc6847a60f360e9ec13a5cc5e0b7 Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 12 Apr 2017 14:48:02 +0200 Subject: [PATCH 45/46] Bring back original README --- README.md | 202 ++++++++++++++++++++++++------------------------------ 1 file changed, 89 insertions(+), 113 deletions(-) diff --git a/README.md b/README.md index 4c97863..548b7fa 100644 --- a/README.md +++ b/README.md @@ -1,140 +1,116 @@ -# asciinema +# asciinema.org -FIXME: description +[![Build Status](https://travis-ci.org/asciinema/asciinema.org.svg?branch=master)](https://travis-ci.org/asciinema/asciinema.org) +[![Code Climate](https://codeclimate.com/github/asciinema/asciinema.org/badges/gpa.svg)](https://codeclimate.com/github/asciinema/asciinema.org) +[![Coverage Status](https://coveralls.io/repos/asciinema/asciinema.org/badge.svg)](https://coveralls.io/r/asciinema/asciinema.org) -## Developing +Record and share your terminal sessions, the right way. -### Setup +asciinema is a free and open source solution for recording terminal sessions +and sharing them on the web. -When you first clone this repository, run: +This is the source code of asciinema.org website. You can find asciinema's +terminal recorder at +[asciinema/asciinema](https://github.com/asciinema/asciinema) and asciinema +player at +[asciinema/asciinema-player](https://github.com/asciinema/asciinema-player). + +## Setup instructions + +Below you'll find setup instructions in case you want to contribute, play with +it on your local machine, or setup your own instance for private use or for +your organization. + +### Quickstart Using Docker Compose + +Required: + - [Docker](https://docs.docker.com/engine/getstarted/step_one/#step-1-get-docker) + - [docker-compose 1.5+](https://docs.docker.com/compose/install/) +```bash +$ wget https://raw.githubusercontent.com/asciinema/asciinema.org/master/docker-compose.yml +$ docker-compose up -d asciinema +$ docker-compose run --rm db_init -```sh -lein setup ``` -This will create files for local configuration, and prep your system -for the project. +You can override the address/port that is sent in email with login token by passing HOST="host:port" environment variable when starting the web server. -### Environment +Assuming you are running Docker Toolbox and VirtualBox: go to http://192.168.99.100:3000/ and enjoy. -To begin developing, start with a REPL. +### Manual setup -```sh -lein repl +#### 1. Install dependencies + +asciinema.org site is a Ruby on Rails application. You need to have following +dependencies installed: + +* Ruby 2.0+ (Ruby 2.1 is recommended) + +* bundler gem + `gem install bundler` + +* PostgreSQL 8+ with libpq development headers + `sudo apt-get install postgresql libpq-dev` on Debian/Ubuntu + +* asciinema's libtsm fork (`asciinema` branch) + See [here](https://github.com/asciinema/libtsm/blob/asciinema/README) for installation instructions. + If you don't install it now the setup script (point 4 below) will try to + install it for you anyway. + +* phantomjs 2.0+ + `sudo add-apt-repository ppa:tanguy-patte/phantomjs && sudo apt-get update && sudo apt-get install phantomjs` + +#### 2. Get the source code + +Clone git repository: + +```bash +$ git clone git://github.com/asciinema/asciinema.org.git +$ cd asciinema.org ``` -Then load the development environment. +#### 3. Prepare database config file -```clojure -user=> (dev) -:loaded +Copy *config/database.yml.example* to *config/database.yml*. Then set +database/user/password to whatever you prefer. + +If database specified in database.yml doesn't exist then the following setup +task will create it (make sure database user is allowed to create new +databases). + +#### 4. Setup the app + +Following script will install gem dependencies and setup database: + +```bash +$ ./script/setup ``` -Run `go` to initiate and start the system. +#### 5. Run Rails server -```clojure -dev=> (go) -:started +```bash +$ bundle exec rails server ``` -By default this creates a web server at . +#### 6. Run the background job processor -When you make changes to your source files, use `reset` to reload any -modified files and reset the server. +The background job processor is needed for asciicast pre-processing and +thumbnail generation. -```clojure -dev=> (reset) -:reloading (...) -:resumed +```bash +$ bundle exec sidekiq ``` -### Testing +## Contributing -Testing is fastest through the REPL, as you avoid environment startup -time. +If you want to contribute to this project check out +[Contributing](http://asciinema.org/contributing) page. -```clojure -dev=> (test) -... -``` +## Authors -But you can also run tests through Leiningen. +Developed with passion by [Marcin Kulik](http://ku1ik.com) and great open +source [contributors](https://github.com/asciinema/asciinema.org/contributors) -```sh -lein test -``` +## Copyright -### Migrations - -Migrations are handled by [ragtime][]. Migration files are stored in -the `resources/migrations` directory, and are applied in alphanumeric -order. - -To update the database to the latest migration, open the REPL and run: - -```clojure -dev=> (migrate) -Applying 20150815144312-create-users -Applying 20150815145033-create-posts -``` - -To rollback the last migration, run: - -```clojure -dev=> (rollback) -Rolling back 20150815145033-create-posts -``` - -Note that the system needs to be setup with `(init)` or `(go)` before -migrations can be applied. - -[ragtime]: https://github.com/weavejester/ragtime - -### Generators - -This project has several generator functions to help you create files. - -To create a new endpoint: - -```clojure -dev=> (gen/endpoint "bar") -Creating file src/foo/endpoint/bar.clj -Creating file test/foo/endpoint/bar_test.clj -Creating directory resources/foo/endpoint/bar -nil -``` - -To create a new component: - -```clojure -dev=> (gen/component "baz") -Creating file src/foo/component/baz.clj -Creating file test/foo/component/baz_test.clj -nil -``` - -To create a new boundary: - -```clojure -dev=> (gen/boundary "quz" foo.component.baz.Baz) -Creating file src/foo/boundary/quz.clj -Creating file test/foo/boundary/quz_test.clj -nil -``` - -To create a new SQL migration: - -```clojure -dev=> (gen/sql-migration "create-users") -Creating file resources/foo/migrations/20160519143643-create-users.up.sql -Creating file resources/foo/migrations/20160519143643-create-users.down.sql -nil -``` - -## Deploying - -FIXME: steps to deploy - -## Legal - -Copyright © 2017 FIXME +Copyright © 2011-2016 Marcin Kulik. See LICENSE.txt for details. From d37bff0613e78ed269124d440a1160adec4cf64a Mon Sep 17 00:00:00 2001 From: Marcin Kulik Date: Wed, 12 Apr 2017 15:43:45 +0200 Subject: [PATCH 46/46] Add missing whitespace --- src/asciinema/model/asciicast.clj | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/asciinema/model/asciicast.clj b/src/asciinema/model/asciicast.clj index 8d28d11..da2bbbb 100644 --- a/src/asciinema/model/asciicast.clj +++ b/src/asciinema/model/asciicast.clj @@ -1,6 +1,6 @@ (ns asciinema.model.asciicast - (:require[pandect.algo.sha1 :as sha1] - [clojure.string :as str])) + (:require [pandect.algo.sha1 :as sha1] + [clojure.string :as str])) (defn json-store-path [{:keys [id file stdout_frames]}] (cond