boot-gae: task library
|
This documentation will use the namespace alias gae to refer to
boot-gae tasks. For example, gae/config means
migae.boot-gae/config .
|
-
appengine
generate gae appengine-web.xml
config file
-
build
assemble, config, compile, etc.
-
deploy
install a new version of the application onto the server
-
filters
generate a "filter-generator" clojure file containing gen-class directives and aot-compile it to generate filter class files
-
install-sdk
unpack and install the SDK zipfile
-
libs
install dependency jars in <build-dir>/WEB-INF/lib
-
logging
generate and install gae logging config files (e.g. logging.properties
, log4j.properties
)
-
monitor
monitor source tree and propagate changes to output dir
-
prod
build for production (without reloader
task)
-
reloader
generate and aot-compile reloader filter used to support reload-on-refresh
-
run
run devappserver
-
servlets
generate a "servlet-generator" clojure file containing gen-class directives and aot-compile it to produce servlet class files
-
webxml
generate gae web.xml
config file
Generates WEB-INF/appengine-web.xml
from appengine.edn
Enables Google’s built-in Appstats service. Configure in appstats.edn
Convenience wrapper composing the tasks required to assemble,
configure, aot-compile, etc.
(boot/deftask build
"assemble, configure, and build app"
[k keep bool "keep intermediate .clj and .edn files"
p prod bool "production build, without reloader"
v verbose bool "verbose"]
(let [keep (or keep false)
verbose (or verbose false)]
;; mod (str (-> (boot/get-env) :gae :module))]
;; (println "MODULE: " mod)
(comp (install-sdk)
(libs :verbose verbose)
(logging :verbose verbose)
(appstats :verbose verbose)
(builtin/javac)
(if prod identity (reloader :keep keep :verbose verbose))
(filters :keep keep :verbose verbose)
(servlets :keep keep :verbose verbose)
(webxml :verbose verbose)
(appengine :verbose verbose)
(builtin/sift :move {#"(.*clj$)" (str classes-dir "/$1")})
(builtin/sift :move {#"(.*\.class$)" (str classes-dir "/$1")})
(builtin/pom)
(builtin/jar)
)))
Some of the tasks can be run in any order (e.g. filters and servlets) but webxml
must be run after servlets
, filters
, appstats
and reloader
, since it uses the web.xml.edn
file they elaborate. The reloader
`filters
, and appstats
tasks are optional.
You will obviously need a GAE account to deploy.
To deploy run $ boot gae/deploy
.
|
If you have multiple google accounts, you may get an error
telling you the app-id is not found, if the deployment code thinks
you are not logged on to the right account. The deployment logic
uses ~/.appcfg_oauth2_tokens_java for this; just delete that file
and try again.
|
Copy all dependency jars to WEB-INF/lib
, required by Appengine.
This is a convenience wrapper around boot tasks (watch
, etc.). It
watches the source tree and copies changed files to the correct
output dir.
(boot/deftask prod
"make a production build, excluding reloader"
[k keep bool "keep intermediate .clj and .edn files"
v verbose bool "verbose"]
(let [keep (or keep false)
verbose (or verbose false)
mod (str (-> (boot/get-env) :gae :module :name))]
(comp (install-sdk)
(libs :verbose verbose)
(logging :verbose verbose)
(appstats :verbose verbose)
(builtin/javac)
(filters :keep false :verbose verbose)
(servlets :keep false :verbose verbose)
(webxml :verbose verbose)
(appengine :verbose verbose)
(builtin/sift :move {#"(.*clj$)" (str classes-dir "/$1")})
(builtin/sift :move {#"(.*\.class$)" (str classes-dir "/$1")})
(builtin/target :dir #{(str "target/" mod)})
)))
Install a servlet filter to support reload-on-refresh in conjunction
with the monitor
task. For use during development, in order to get
repl-like responsiveness. Run gae/reloader -k
to see the intermediate files.
Add security-constraint stanzas to web.xml
. Configuration data goes in security.edn
.
Servlets can be implemented in Clojure in a variety of ways; boot-gae
supports the technique described here out of the box, via the
gae/servlets
task. That task reads the gae
configuration map from
build.boot
, processes a stencil template file to generate a Clojure
source file, and then aot compiles that source file.
|
the following config example is OBSOLETE - boot-gae now uses
.edn files for configuration. But the processing is pretty much as
described here. See the greetings-gae test app for examples. Try
running gae/servlets with and without the :keep flag to see the
intermediate files.
|
|
Tasks like filters , servlets , and reloader work by
elaborating web.xml.edn . They don’t actually produce output; that’s
the job of the webxml task that uses the web.xml.edn config info
to generate the actual WEB-INF/web.xml file.
|
Here’s an example: this configuration map:
build.boot
(def gae
{ ...
:servlet-ns 'migae.servlets
:servlets [{:ns 'migae.echo ;; = servlet-class
;; :jsp - alternative to :ns, for using java servlet pages
:name "echo-servlet"
:display {:name "Awesome Echo Servlet"}
:desc {:text "blah blah"}
:url "/echo/*"
:params [{:name "greeting" :val "Hello"}]
:load-on-startup {:order 3}}
{:ns 'migae.math ;; REQUIRED
:name "math-servlet" ;; REQUIRED
:url "/math/*" ;; REQUIRED
:params [{:name "op" :val "+"}
{:name "arg1" :val 3}
{:name "arg2" :val 2}]}]
...}
will produce the following Clojure:
migae/servlets.clj
(ns migae.servlets)
(gen-class :name migae.echo
:extends javax.servlet.http.HttpServlet
:impl-ns migae.echo)
(gen-class :name migae.math
:extends javax.servlet.http.HttpServlet
:impl-ns migae.math)
(gen-class :name migae.reloader
:implements [javax.servlet.Filter]
:impl-ns migae.reloader)
|
The namespace for this file is specified by the :servlet-ns key of
the gae config map, and the gen-class :name and :impl-ns values
are from the :servlets key. See the example below.
Note that :servlets-ns is a little misleading; the generated file is
not itself a servlet, it’s just there to hold the gen-class
operations that generate the actual servlet code.
|
|
By default the generated clojure file will be discarded once it has been aot compiled. To save it, add the
|
Servlet implementations will look like this:
migae/echo.clj
(ns migae.echo
(:require [clojure.math.numeric-tower :as math]
[compojure.core :refer :all]
[compojure.route :as route]
[ring.handler.dump :refer :all] ; ring-devel
[ring.util.response :as rsp]
[ring.util.servlet :as servlet]
[ring.middleware.params :refer [wrap-params]] ; in ring-core
[ring.middleware.defaults :refer :all])) ; ring-defaults
(defroutes echo-routes
(context "/echo" []
(GET "/hello/:name" [name]
(-> (rsp/response (str "Hello there, " name))
(rsp/content-type "text/html")))
(route/not-found "<h1>Echo API not found</h1>")))
(servlet/defservice
(-> (routes
echo-routes)
(wrap-defaults api-defaults)
))
Generates WEB-INF/web.xml
from web.xml.edn
. Run this task after
you run servlets
, filters
, reloader
, and appstats
.
|
This section is somewhat out of date. Now the tasks
gae/monitor and gae/reloader together give you the technique
described below out of the box.
|
Then you’ll need to prepare things, as described below. Once that’s
done, you’ll no doubt want to do repl-based development: change some
source code and have the results show up immediately in the browser.
We’re not quite that replish: you have to refresh the browser. To
make this work, you have to copy your changes into WEB-INF/classes
.
That’s because the GAE dev server will refuse to look anywhere else
for resources, for security reasons.
So if you want repl, you need to do two things. First, run the
following command before you start editing:
Now when a source file changes, it will be copied to the corresponding
output directory, and the reloader
filter will reload the changed
namespace.
|
If you’re going to be working with multiple file types you’ll need to adjust the regex and/or run multiple monitor pipelines.
For example, if you want to edit .css files located at <approot> (that
is, not in WEB-INF), you would run something like:
$ boot monitor sift -i "html$" target -d "build" -C
|
The second thing you need to do is install a filter servlet that will
reload your Clojure files from WEB-INF/classes
. Here’s an example:
|
OBSOLETE. The basic logic described here still applies;
however, boot-migae takes care of all of this automatically. You do
not need to configure anything, just run the reloader task. If you
run gae/build -k you will see the reloader.clj file described below.
|
migae/reloader.clj
(ns migae.reloader
(:import (javax.servlet Filter FilterChain FilterConfig
ServletRequest ServletResponse))
(:require [ns-tracker.core :refer :all]))
(defn -init [^Filter this ^FilterConfig cfg])
(defn -destroy [^Filter this])
(def modified-namespaces (ns-tracker ["./"]))
(defn -doFilter
[^Filter this
^ServletRequest rqst
^ServletResponse resp
^FilterChain chain]
(doseq [ns-sym (modified-namespaces)]
(require ns-sym :reload))
(.doFilter chain rqst resp))
The gae/webxml
task will pick up the info from gae/reloader
and
generate the appropriate stanza for the web.xml
file:
WEB-INF/web.xml
<filter-mapping>
<url-pattern>/echo/*</url-pattern>
<filter-name>reloader</filter-name>
</filter-mapping>
<filter-mapping>
<url-pattern>/math/*</url-pattern>
<filter-name>reloader</filter-name>
</filter-mapping>
<filter>
<filter-name>reloader</filter-name>
<filter-class>migae.reloader</filter-class>
</filter>
To make this work, all you need to provide is the migae/reloader.clj
file and set the configuration map. The WEB-INF/web.xml
file will
be autogenned as explained in [config], and the
migae/servlets.clj
file will be autogenned, aot-compiled, and
discarded, as explained in servlets.
Once your build.boot
is set up, you need to prepared the system.
boot-gae has a dependency on the GAE sdk, so the first time you run it
it will be downloaded. Don’t be alarmed if it takes a while; the SDK
is a ~165 MB zipfile.
The GAE dev server requires that the SDK be available in exploded
form. The maven artifact that gets installed into ~/.m2/repository
is a zipfile; the gae/install-sdk
task will explode it and install
it.
Use the :sdk-root
key in the gae
configuration map to specify a
location. The default is ~/.appengine-sdk
; if you want to be
compatible with Gradle, use `:sdk-root "~/.gradle/appengine-sdk".
Once the SDK is installed, proceed with preparing your webapp. GAE
has strict security rules; the dev server will not allow access to
anything outside of the webapp’s root directory. That means that
everything that needs to be on the classpath must be installed in
<approot>/WEB-INF
. For libraries that means all the jarfile
dependencies must be copied into <approot>/WEB-INF/lib
. The
gae/libs
task takes care of this:
$ boot gae/libs
Adding uberjar entries...
Sifting output files...
Writing target dir(s)...
Now you have four tasks remaining:
-
copy sources/resources into the build tree so they will be accessible by the dev server
-
configure logging - gae/logging
; configuration is set via the :logging
key in the config map
-
configure appengine and the servlet container (create appengine-web.xml and web.xml)
-
aot compile your servlets - servlets does this.
|
The way boot works is that the target task will copy stuff to the
build directory. So for example, if you have foo.html at the root
of your resources dir, target will put it in the same place
relative to the build dir, so it will end up in <build-dir>/ . For
static assets that’s generally a good thing.
|
For Clojure files, and for anything that you want to move into
WEB-INF
(thereby removing it from public accessibility), you need to
use the sift
task instead. In particular the :move
parameter to
sift
allows you to pick out the files you are interested in and rewrite
their paths.
You could use sift
to arrange things by hand, but as a convenience
the gae/clj
task will do it for you.
|
Waaaay obsolete. We now use .edn config files. But the
map structure is pretty much the same. See the sample app
greetings-gae for examples.
|
The configuration map is used by the gae/config
task to generate the
web.xml
and appengine-web.xml
files required by GAE.
It is also used by the gae/servlets
task, which generates and aot
compiles the Clojure code needed to support servlet development; see
servlets for details.
(def gae
;; https://cloud.google.com/appengine/docs/java/config/webxml
;; web.xml doco: http://docs.oracle.com/cd/E13222_01/wls/docs81/webapp/web_xml.html
{;; :build-dir ; default: "build"; gradle compatibility: "build/exploded-app"
;; :sdk-root ; default: ~/.appengine-sdk; gradle compatibility: "~/.gradle/appengine-sdk"
:list-tasks true ;; print "TASK: <taskname>"
;; :verbose true
:aot #{'migae.servlets}
:app-id (clojure.string/replace +project+ #"/" ".")
:module "foo"
;; gae version string syntax: no '.', lowercase only, etc
:version (clojure.string/lower-case (clojure.string/replace +version+ #"\." "-"))
:display-name {:name "hello app"} ;; web.xml <display-name>
:descr {:text "description of this web app, for web.xml etc."} ;; web.xml
;; appengine-web.xml: see https://cloud.google.com/appengine/docs/java/config/appconfig
:appengine {:thread-safe true
;; :public-root "/static"
:system-properties {:props [{:name "myapp.maximum-message-length" :value "140"}
{:name "myapp.notify-every-n-signups" :value "1000"}
{:name"myapp.notify-url"
:value "http://www.example.com/supnotfy"}]}
;; :env-vars [{:name "FOO" :value "BAR"}]
:logging {:jul {:name "java.util.logging.config.file"
:value "WEB-INF/logging.properties"}}
;; #_{:log4j {:name "java.util.logging.config.file"
;; :value "WEB-INF/classes/log4j.properties"}}}
:sessions true
:ssl true
:async-session-persistence {:enabled "true" :queue-name "myqueue"}
:inbound-services [{:service :mail} {:service :warmup}]
:precompilation true
;; :scaling {:basic {:max-instances 11 :idle-timeout "10m"
;; :instance-class "B2"}
;; :manual {:instances 5
;; :instance-class "B2"}
;; :automatic {:instance-class "F2"
;; :idle-instances {:min 5
;; ;; ‘automatic’ is the default value.
;; :max "automatic"}
;; :pending-latency {:min "30ms" :max "automatic"}
;; :concurrent-requests {:max 50}}}
;; :resource-files {:include [{:path "**.xml"
;; :expiration "4d h5"
;; :http-header {:name "Access-Control-Allow-Origin"
;; :value "http://example.org"}}]
;; :exclude [{:path "feed/**.xml"}]}
;; :static-files {:include {:path "foo/**.png"
;; :expiration "4d h5"
;; :http-header {:name "Access-Control-Allow-Origin"
;; :value "http://example.org"}}
;; :exclude {:path "bar/**.zip"}}
}
:welcome {:file "index.html"}
:errors [{:code 404 :url "/404.html"}] ;; use :code, or:type, e.g 'java.lang.String
;;mime: see http://www.opensource.apple.com/source/JBoss/JBoss-739/jakarta-tomcat-LE-jdk14/conf/web.xml
:mime-mappings [{:ext "abs" :type "audio/x-mpeg"}
{:ext "gz" :type "application/x-gzip"}
{:ext "htm" :type "text/html"}
{:ext "html" :type "text/html"}
{:ext "svg" :type "image/svg+xml"}
{:ext "txt" :type "text/plain"}
{:ext "xml" :type "text/xml"}
{:ext "xsl" :type "text/xsl"}
{:ext "zip" :type "application/zip"}]
;; servlet config: the config task will:
:servlet-ns 'migae.servlets ;; autogen migae/servlets.clj from a stencil template
;; :servlets used to gen :servlet-ns file AND servlet configs in web.xml
:servlets [{:ns 'migae.echo ;; web.xml <servlet-class>
:name "echo-servlet" ;; REQUIRED
:url "/echo/*" ;; REQUIRED
:display {:name "Awesome Echo Servlet"} ;; web.xml <display-name>
:desc {:text "description of this servlet blah blah"}
:params [{:name "greeting" :val "Hello"}]
:load-on-startup {:order 3}}
{:ns 'migae.math
:name "math-servlet"
:url "/math/*"
:params [{:name "op" :val "+"}
{:name "arg1" :val 3}
{:name "arg2" :val 2}]}]
;; appstats is specific to GAE
;; see https://cloud.google.com/appengine/docs/java/tools/appstats
:appstats {:admin-console {:url "/appstats" :name "Appstats"}
:name "appstats"
:desc {:text "Google Appstats Service"}
:url "/admin/appstats/*"
:security-role "admin"
:filter {:display {:name "Google Appstats"}
:desc {:text "Google Appstats Filter"}
:url "/*"
:params [{:name "logMessage"
:val "Appstats available: /appstats/details?time={ID}"}
{:name "calculateRpcCosts"
:val true}]}
:servlet {:display {:name "Google Appstats"}}}
;; if you want a repl-like environment on the dev server,
;; you must use a servlet filter to reload your clojure code
;; see http://www.oracle.com/technetwork/java/filters-137243.html
:filters [{:ns 'migae.reloader ; REQUIRED
:name "reloader" ; REQUIRED
:display {:name "Clojure reload filter"} ; OPTIONAL
:urls [{:url "echo/*"}
{:url "math/*"}]
:desc {:text "clojure reload filter"}}]
;; web.xml security constraints
;; see http://docs.oracle.com/javaee/5/tutorial/doc/bncbe.html
;;
:security [{:resource {:name "foo" :desc {:text "Foo resource security"}
:url "/foo/*"}
:role "admin"}]})
== gae programming with clojure
You know about the whitelist. Did you notice the fine print?
|
Just because a class is whitelisted doesn’t mean that all the
features and operations of the class are supported for an app running
in the App Engine sandbox environment.
|
For example, this will fail with an access exception:
(let [fac (javax.xml.stream.XMLInputFactory/newFactory)
sr (java.io.StringReader "foo")
xmlsreader (.createXMLStreamReader fac sr)]
That’s because this call to .createXMLStreamReader
cannot be
resolved at compile time, so at runtime Clojure will try to use
reflection to invoke the method. The involves a call to getMethods
that GAE disallows.
To fix this you need to provide a type hint so that Clojure can
resolve the call at compile time:
(.createXMLStreamReader fac ^StringReader sr)