Liking cljdoc? Tell your friends :D

boot-gae: task library

This documentation will use the namespace alias gae to refer to boot-gae tasks. For example, gae/config means migae.boot-gae/config.

  • appengine generate gae appengine-web.xml config file

  • build assemble, config, compile, etc.

  • deploy install a new version of the application onto the server

  • filters generate a "filter-generator" clojure file containing gen-class directives and aot-compile it to generate filter class files

  • install-sdk unpack and install the SDK zipfile

  • libs install dependency jars in <build-dir>/WEB-INF/lib

  • logging generate and install gae logging config files (e.g. logging.properties, log4j.properties)

  • monitor monitor source tree and propagate changes to output dir

  • prod build for production (without reloader task)

  • reloader generate and aot-compile reloader filter used to support reload-on-refresh

  • run run devappserver

  • servlets generate a "servlet-generator" clojure file containing gen-class directives and aot-compile it to produce servlet class files

  • webxml generate gae web.xml config file

appengine

Generates WEB-INF/appengine-web.xml from appengine.edn

appstats

Enables Google’s built-in Appstats service. Configure in appstats.edn

build

Convenience wrapper composing the tasks required to assemble, configure, aot-compile, etc.

The build pipeline:

(boot/deftask build
  "assemble, configure, and build app"
  [k keep bool "keep intermediate .clj and .edn files"
   p prod bool "production build, without reloader"
   v verbose bool "verbose"]
  (let [keep (or keep false)
        verbose (or verbose false)]
    ;;     mod (str (-> (boot/get-env) :gae :module))]
    ;; (println "MODULE: " mod)
    (comp (install-sdk)
          (libs :verbose verbose)
          (logging :verbose verbose)
          (appstats :verbose verbose)
          (builtin/javac)
          (if prod identity (reloader :keep keep :verbose verbose))
          (filters :keep keep :verbose verbose)
          (servlets :keep keep :verbose verbose)
          (webxml :verbose verbose)
          (appengine :verbose verbose)
          (builtin/sift :move {#"(.*clj$)" (str classes-dir "/$1")})
          (builtin/sift :move {#"(.*\.class$)" (str classes-dir "/$1")})
          (builtin/pom)
          (builtin/jar)
          )))

Some of the tasks can be run in any order (e.g. filters and servlets) but webxml must be run after servlets, filters, appstats and reloader, since it uses the web.xml.edn file they elaborate. The reloader `filters, and appstats tasks are optional.

deploy

You will obviously need a GAE account to deploy.

To deploy run $ boot gae/deploy.

If you have multiple google accounts, you may get an error telling you the app-id is not found, if the deployment code thinks you are not logged on to the right account. The deployment logic uses ~/.appcfg_oauth2_tokens_java for this; just delete that file and try again.

libs

Copy all dependency jars to WEB-INF/lib, required by Appengine.

monitor

This is a convenience wrapper around boot tasks (watch, etc.). It watches the source tree and copies changed files to the correct output dir.

prod

(boot/deftask prod
  "make a production build, excluding reloader"
  [k keep bool "keep intermediate .clj and .edn files"
   v verbose bool "verbose"]
  (let [keep (or keep false)
        verbose (or verbose false)
        mod (str (-> (boot/get-env) :gae :module :name))]
    (comp (install-sdk)
          (libs :verbose verbose)
          (logging :verbose verbose)
          (appstats :verbose verbose)
          (builtin/javac)
          (filters :keep false :verbose verbose)
          (servlets :keep false :verbose verbose)
          (webxml :verbose verbose)
          (appengine :verbose verbose)
          (builtin/sift :move {#"(.*clj$)" (str classes-dir "/$1")})
          (builtin/sift :move {#"(.*\.class$)" (str classes-dir "/$1")})
          (builtin/target :dir #{(str "target/" mod)})
          )))

reloader

Install a servlet filter to support reload-on-refresh in conjunction with the monitor task. For use during development, in order to get repl-like responsiveness. Run gae/reloader -k to see the intermediate files.

run

security

Add security-constraint stanzas to web.xml. Configuration data goes in security.edn.

not yet implemented

servlets

Servlets can be implemented in Clojure in a variety of ways; boot-gae supports the technique described here out of the box, via the gae/servlets task. That task reads the gae configuration map from build.boot, processes a stencil template file to generate a Clojure source file, and then aot compiles that source file.

the following config example is OBSOLETE - boot-gae now uses .edn files for configuration. But the processing is pretty much as described here. See the greetings-gae test app for examples. Try running gae/servlets with and without the :keep flag to see the intermediate files.
Tasks like filters, servlets, and reloader work by elaborating web.xml.edn. They don’t actually produce output; that’s the job of the webxml task that uses the web.xml.edn config info to generate the actual WEB-INF/web.xml file.

Here’s an example: this configuration map:

build.boot
(def gae
{ ...
   :servlet-ns 'migae.servlets
   :servlets [{:ns 'migae.echo  ;; = servlet-class
               ;; :jsp - alternative to :ns, for using java servlet pages
               :name "echo-servlet"
               :display {:name "Awesome Echo Servlet"}
               :desc {:text "blah blah"}
               :url "/echo/*"
               :params [{:name "greeting" :val "Hello"}]
               :load-on-startup {:order 3}}

              {:ns 'migae.math      ;; REQUIRED
               :name "math-servlet"  ;; REQUIRED
               :url "/math/*"      ;; REQUIRED
               :params [{:name "op" :val "+"}
                        {:name "arg1" :val 3}
                        {:name "arg2" :val 2}]}]
...}

will produce the following Clojure:

migae/servlets.clj
(ns migae.servlets)

(gen-class :name migae.echo
           :extends javax.servlet.http.HttpServlet
           :impl-ns migae.echo)
(gen-class :name migae.math
           :extends javax.servlet.http.HttpServlet
           :impl-ns migae.math)

(gen-class :name migae.reloader
           :implements [javax.servlet.Filter]
           :impl-ns migae.reloader)

The namespace for this file is specified by the :servlet-ns key of the gae config map, and the gen-class :name and :impl-ns values are from the :servlets key. See the example below.

Note that :servlets-ns is a little misleading; the generated file is not itself a servlet, it’s just there to hold the gen-class operations that generate the actual servlet code.

By default the generated clojure file will be discarded once it has been aot compiled. To save it, add the

Servlet implementations will look like this:

migae/echo.clj
(ns migae.echo
  (:require [clojure.math.numeric-tower :as math]
            [compojure.core :refer :all]
            [compojure.route :as route]
            [ring.handler.dump :refer :all] ; ring-devel
            [ring.util.response :as rsp]
            [ring.util.servlet :as servlet]
            [ring.middleware.params :refer [wrap-params]] ; in ring-core
            [ring.middleware.defaults :refer :all])) ; ring-defaults

(defroutes echo-routes
  (context "/echo" []
    (GET "/hello/:name" [name]
         (-> (rsp/response (str "Hello there, " name))
             (rsp/content-type "text/html")))
    (route/not-found "<h1>Echo API not found</h1>")))

(servlet/defservice
   (-> (routes
        echo-routes)
       (wrap-defaults api-defaults)
       ))

webxml

Generates WEB-INF/web.xml from web.xml.edn. Run this task after you run servlets, filters, reloader, and appstats.

replry

This section is somewhat out of date. Now the tasks gae/monitor and gae/reloader together give you the technique described below out of the box.

Then you’ll need to prepare things, as described below. Once that’s done, you’ll no doubt want to do repl-based development: change some source code and have the results show up immediately in the browser. We’re not quite that replish: you have to refresh the browser. To make this work, you have to copy your changes into WEB-INF/classes. That’s because the GAE dev server will refuse to look anywhere else for resources, for security reasons.

So if you want repl, you need to do two things. First, run the following command before you start editing:

$ boot gae/monitor

Now when a source file changes, it will be copied to the corresponding output directory, and the reloader filter will reload the changed namespace.

FIXME: obsolete

If you’re going to be working with multiple file types you’ll need to adjust the regex and/or run multiple monitor pipelines.

For example, if you want to edit .css files located at <approot> (that is, not in WEB-INF), you would run something like:

$ boot monitor sift -i "html$" target -d "build" -C

The second thing you need to do is install a filter servlet that will reload your Clojure files from WEB-INF/classes. Here’s an example:

OBSOLETE. The basic logic described here still applies; however, boot-migae takes care of all of this automatically. You do not need to configure anything, just run the reloader task. If you run gae/build -k you will see the reloader.clj file described below.
migae/reloader.clj
(ns migae.reloader
  (:import (javax.servlet Filter FilterChain FilterConfig
                          ServletRequest ServletResponse))
  (:require [ns-tracker.core :refer :all]))
(defn -init [^Filter this ^FilterConfig cfg])
(defn -destroy [^Filter this])
(def modified-namespaces (ns-tracker ["./"]))
(defn -doFilter
  [^Filter this
   ^ServletRequest rqst
   ^ServletResponse resp
   ^FilterChain chain]
  (doseq [ns-sym (modified-namespaces)]
    (require ns-sym :reload))
  (.doFilter chain rqst resp))

The gae/webxml task will pick up the info from gae/reloader and generate the appropriate stanza for the web.xml file:

WEB-INF/web.xml
  <filter-mapping>
    <url-pattern>/echo/*</url-pattern>
    <filter-name>reloader</filter-name>
  </filter-mapping>
  <filter-mapping>
    <url-pattern>/math/*</url-pattern>
    <filter-name>reloader</filter-name>
  </filter-mapping>
  <filter>
    <filter-name>reloader</filter-name>
    <filter-class>migae.reloader</filter-class>
  </filter>

To make this work, all you need to provide is the migae/reloader.clj file and set the configuration map. The WEB-INF/web.xml file will be autogenned as explained in [config], and the migae/servlets.clj file will be autogenned, aot-compiled, and discarded, as explained in servlets.

how it works

Once your build.boot is set up, you need to prepared the system. boot-gae has a dependency on the GAE sdk, so the first time you run it it will be downloaded. Don’t be alarmed if it takes a while; the SDK is a ~165 MB zipfile.

The GAE dev server requires that the SDK be available in exploded form. The maven artifact that gets installed into ~/.m2/repository is a zipfile; the gae/install-sdk task will explode it and install it.

Use the :sdk-root key in the gae configuration map to specify a location. The default is ~/.appengine-sdk; if you want to be compatible with Gradle, use `:sdk-root "~/.gradle/appengine-sdk".

Once the SDK is installed, proceed with preparing your webapp. GAE has strict security rules; the dev server will not allow access to anything outside of the webapp’s root directory. That means that everything that needs to be on the classpath must be installed in <approot>/WEB-INF. For libraries that means all the jarfile dependencies must be copied into <approot>/WEB-INF/lib. The gae/libs task takes care of this:

$ boot gae/libs
Adding uberjar entries...
Sifting output files...
Writing target dir(s)...

Now you have four tasks remaining:

  • copy sources/resources into the build tree so they will be accessible by the dev server

  • configure logging - gae/logging; configuration is set via the :logging key in the config map

  • configure appengine and the servlet container (create appengine-web.xml and web.xml)

  • aot compile your servlets - servlets does this.

The way boot works is that the target task will copy stuff to the build directory. So for example, if you have foo.html at the root of your resources dir, target will put it in the same place relative to the build dir, so it will end up in <build-dir>/. For static assets that’s generally a good thing.

For Clojure files, and for anything that you want to move into WEB-INF (thereby removing it from public accessibility), you need to use the sift task instead. In particular the :move parameter to sift allows you to pick out the files you are interested in and rewrite their paths.

You could use sift to arrange things by hand, but as a convenience the gae/clj task will do it for you.

Waaaay obsolete. We now use .edn config files. But the map structure is pretty much the same. See the sample app greetings-gae for examples.

The configuration map is used by the gae/config task to generate the web.xml and appengine-web.xml files required by GAE.

It is also used by the gae/servlets task, which generates and aot compiles the Clojure code needed to support servlet development; see servlets for details.

See

Example:

(def gae
  ;; https://cloud.google.com/appengine/docs/java/config/webxml
  ;; web.xml doco: http://docs.oracle.com/cd/E13222_01/wls/docs81/webapp/web_xml.html
  {;; :build-dir ; default: "build";  gradle compatibility: "build/exploded-app"
   ;; :sdk-root ; default: ~/.appengine-sdk; gradle compatibility: "~/.gradle/appengine-sdk"
   :list-tasks true ;; print "TASK: <taskname>"
   ;; :verbose true
   :aot #{'migae.servlets}
   :app-id (clojure.string/replace +project+ #"/" ".")
   :module "foo"
   ;; gae version string syntax: no '.', lowercase only, etc
   :version (clojure.string/lower-case (clojure.string/replace +version+ #"\." "-"))
   :display-name {:name "hello app"}  ;; web.xml <display-name>
   :descr {:text "description of this web app, for web.xml etc."} ;; web.xml
   ;; appengine-web.xml: see https://cloud.google.com/appengine/docs/java/config/appconfig
   :appengine {:thread-safe true
               ;; :public-root "/static"
               :system-properties {:props [{:name "myapp.maximum-message-length" :value "140"}
                                           {:name "myapp.notify-every-n-signups" :value "1000"}
                                           {:name"myapp.notify-url"
                                            :value "http://www.example.com/supnotfy"}]}
               ;; :env-vars [{:name "FOO" :value "BAR"}]
               :logging {:jul {:name "java.util.logging.config.file"
                               :value "WEB-INF/logging.properties"}}
               ;; #_{:log4j {:name "java.util.logging.config.file"
               ;;          :value "WEB-INF/classes/log4j.properties"}}}
               :sessions true
               :ssl true
               :async-session-persistence {:enabled "true" :queue-name "myqueue"}
               :inbound-services [{:service :mail} {:service :warmup}]
               :precompilation true
               ;; :scaling {:basic {:max-instances 11 :idle-timeout "10m"
               ;;                   :instance-class "B2"}
               ;;           :manual {:instances 5
               ;;                    :instance-class "B2"}
               ;;           :automatic {:instance-class "F2"
               ;;                       :idle-instances {:min 5
               ;;                                        ;; ‘automatic’ is the default value.
               ;;                                        :max "automatic"}
               ;;                       :pending-latency {:min "30ms" :max "automatic"}
               ;;                       :concurrent-requests {:max 50}}}
               ;; :resource-files {:include [{:path "**.xml"
               ;;                            :expiration "4d h5"
               ;;                            :http-header {:name "Access-Control-Allow-Origin"
               ;;                                          :value "http://example.org"}}]
               ;;                  :exclude [{:path "feed/**.xml"}]}
               ;; :static-files {:include {:path "foo/**.png"
               ;;                          :expiration "4d h5"
               ;;                          :http-header {:name "Access-Control-Allow-Origin"
               ;;                                        :value "http://example.org"}}
               ;;                :exclude {:path "bar/**.zip"}}
               }
   :welcome {:file "index.html"}
   :errors [{:code 404 :url "/404.html"}] ;; use :code, or:type, e.g 'java.lang.String
   ;;mime: see http://www.opensource.apple.com/source/JBoss/JBoss-739/jakarta-tomcat-LE-jdk14/conf/web.xml
   :mime-mappings [{:ext "abs" :type "audio/x-mpeg"}
   		  {:ext "gz"  :type "application/x-gzip"}
		  {:ext "htm" :type "text/html"}
		  {:ext "html" :type "text/html"}
		  {:ext "svg" :type "image/svg+xml"}
		  {:ext "txt" :type "text/plain"}
		  {:ext "xml" :type "text/xml"}
		  {:ext "xsl" :type "text/xsl"}
		  {:ext "zip" :type "application/zip"}]
   ;; servlet config: the config task will:
   :servlet-ns 'migae.servlets ;; autogen migae/servlets.clj from a stencil template
   ;; :servlets used to gen :servlet-ns file AND servlet configs in web.xml
   :servlets [{:ns 'migae.echo  ;;  web.xml <servlet-class>
               :name "echo-servlet"  ;; REQUIRED
               :url "/echo/*"   ;; REQUIRED
               :display {:name "Awesome Echo Servlet"} ;; web.xml <display-name>
               :desc {:text "description of this servlet blah blah"}
               :params [{:name "greeting" :val "Hello"}]
               :load-on-startup {:order 3}}

              {:ns 'migae.math
               :name "math-servlet"
               :url "/math/*"
               :params [{:name "op" :val "+"}
                        {:name "arg1" :val 3}
                        {:name "arg2" :val 2}]}]
   ;; appstats is specific to GAE
   ;; see https://cloud.google.com/appengine/docs/java/tools/appstats
   :appstats {:admin-console {:url "/appstats" :name "Appstats"}
              :name "appstats"
              :desc {:text "Google Appstats Service"}
              :url "/admin/appstats/*"
              :security-role "admin"
              :filter {:display {:name "Google Appstats"}
                       :desc {:text "Google Appstats Filter"}
                       :url "/*"
                       :params [{:name "logMessage"
                                 :val "Appstats available: /appstats/details?time={ID}"}
                                {:name "calculateRpcCosts"
                                 :val true}]}
              :servlet {:display {:name "Google Appstats"}}}
   ;; if you want a repl-like environment on the dev server,
   ;; you must use a servlet filter to reload your clojure code
   ;; see http://www.oracle.com/technetwork/java/filters-137243.html
   :filters [{:ns 'migae.reloader   ; REQUIRED
              :name "reloader"      ; REQUIRED
              :display {:name "Clojure reload filter"} ; OPTIONAL
              :urls [{:url "echo/*"}
                     {:url "math/*"}]
              :desc {:text "clojure reload filter"}}]
   ;; web.xml security constraints
   ;; see http://docs.oracle.com/javaee/5/tutorial/doc/bncbe.html
   ;;
   :security [{:resource {:name "foo" :desc {:text "Foo resource security"}
                          :url "/foo/*"}
               :role "admin"}]})

== gae programming with clojure

You know about the whitelist. Did you notice the fine print?

Just because a class is whitelisted doesn’t mean that all the features and operations of the class are supported for an app running in the App Engine sandbox environment.

For example, this will fail with an access exception:

(let [fac (javax.xml.stream.XMLInputFactory/newFactory)
      sr (java.io.StringReader "foo")
      xmlsreader (.createXMLStreamReader fac sr)]

That’s because this call to .createXMLStreamReader cannot be resolved at compile time, so at runtime Clojure will try to use reflection to invoke the method. The involves a call to getMethods that GAE disallows.

To fix this you need to provide a type hint so that Clojure can resolve the call at compile time:

(.createXMLStreamReader fac ^StringReader sr)

== TODO

Can you improve this documentation?Edit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

× close