[etaoin "1.1.42"]
:host
for WebDriver:port
for WebDriver:webdriver-url
:path-driver
to WebDriver binary:args-driver
for WebDriver binary:webdriver-failed-launch-retries
:path-browser
to web browser binary:args
for web browser binary:log-level
for the web browser console:driver-log-level
:log-stdout
& :log-stderr
for WebDriver output:driver-log-file
for discovered WebDriver log file:post-stop-fns
to hook in behaviour at driver stop:profile
path to web browser profile:env
variables for WebDriver process:size
of intial web browser window:url
to open in web browser:user-agent
for web browser:download-dir
for web browser:headless
web browser:prefs
for web browser:proxy
for web browser:load-strategy
:capabilities
Etaoin offers the Clojure community a simple way to script web browser interactions from Clojure and Babashka. It is a thin abstraction atop the W3C WebDriver protocol that also endeavors to resolve real-world nuances and implementation differences.
Ivan Grishaev (@igrishaev) created Etaoin and published its first release to Clojars in February 2017. He and his band of faithful contributors grew Etaoin into a well-respected goto-library for browser automation.
If Etaoin is not your cup of tea, you might also consider:
Clojure based:
Wally - A Clojure Playwright wrapper
clj-chrome-devtools - Clojure API for controlling a Chrome DevTools remote
Other:
Selenium - A browser automation framework and ecosystem
Playwright - Reliable end-to-end testing for modern web apps
Puppeteer - A high-level API to control Chrome/Chromium over the DevTools Protocol
Etaoin’s test suite covers the following OSes and browsers for both Clojure and Babashka:
OS | Chrome | Firefox | Safari | Edge |
---|---|---|---|---|
Linux (ubuntu) | yes | yes | - | - |
macOS 1 | yes | yes | yes | yes |
Windows | yes | yes | - | yes |
Our GitHub Actions macOS tests run on silicon (aka arm64, aarch64, or M*) hardware
There are two steps to installation:
Add the etaoin
library as a dependency to your project
Install the WebDriver for each web browser that you want to control with Etaoin
Etaoin supports Clojure v1.10 and above on JDK11 and above.
Add the following into the :dependencies
vector in your project.clj
file:
[etaoin "1.1.42"]
Or the following under :deps
in your deps.edn
file:
etaoin/etaoin {:mvn/version "1.1.42"}
We recommend the current release of babashka.
Add the following under :deps
to your bb.edn
file:
etaoin/etaoin {:mvn/version "1.1.42"}
Babashka uses timbre for logging. Timbre’s default logging level is debug. For a quieter Etaoin experience when using babashka, set the timbre default log level to info:
|
Etaoin controls web browsers via their WebDrivers. Each browser has its own WebDriver implementation that must be installed.
If it is not already installed, you will need to install the web browser (Chrome, Firefox, Edge). This is usually done via a download from the browser’s official site. Safari comes bundled with macOS. |
WebDrivers and browsers are updated regularly to fix bugs. Use current versions. |
Some ways to install WebDrivers:
Google Chrome Driver
macOS: brew install chromedriver
Windows: scoop install chromedriver
Download: Official Chromedriver download
Geckodriver for Firefox
macOS: brew install geckodriver
Windows: scoop install geckodriver
Download: Official geckodriver release page
Safari Driver
macOS only: Set up Safari options as the Webkit page instructs (scroll down to "Running the Example in Safari" section).
Microsoft Edge Driver
macOS: (download manually)
Windows: scoop install edgedriver
Edge and msedgedriver
must match, so you might need to specify the version:
scoop install edgedriver@101.0.1210.0
Download: Official Microsoft download site
Check your WebDriver installations by launching these commands. Each should start a process that includes its own local HTTP server. Use Ctrl-C to terminate.
chromedriver
geckodriver
safaridriver -p 0
msedgedriver
You can optionally run the Etaoin test suite to verify your installation.
Some Etaoin API tests rely on ImageMagick. Install it before running tests. |
From a clone of the Etaoin GitHub repo:
To check tools of interest to Etaoin:
bb tools-versions
Run all tests:
bb test:bb
For a smaller sanity test, you might want to run api tests against browsers you are particularly interested in. Example:
bb test:bb --suites api --browsers chrome
During the test run, browser windows will open and close in series. The tests use a local handcrafted HTML file to validate most interactions.
See Troubleshooting if you have problems - or reach out on Clojurians Slack #etaoin or GitHub issues.
The great news is that you can automate your browser directly from your Babashka or Clojure REPL. Let’s interact with Wikipedia:
(require '[etaoin.api :as e]
'[etaoin.keys :as k]
'[clojure.string :as str])
;; Start WebDriver for Firefox
(def driver (e/firefox)) ;; a Firefox window should appear
(e/driver-type driver)
;; => :firefox
;; let's perform a quick Wiki session
;; navigate to Wikipedia
(e/go driver "https://en.wikipedia.org/")
;; make sure we aren't using a large screen layout
(e/set-window-size driver {:width 1280 :height 800})
;; wait for the search input to load
(e/wait-visible driver [{:tag :input :name :search}])
;; search for something interesting
(e/fill driver {:tag :input :name :search} "Clojure programming language")
(e/wait driver 1)
(e/fill driver {:tag :input :name :search} k/enter)
(e/wait-visible driver {:class :mw-search-results})
;; click on first match
(e/click driver [{:class :mw-search-results} {:class :mw-search-result-heading} {:tag :a}])
(e/wait-visible driver {:id :firstHeading})
;; check our new url location
;; (wikipedia can tack on a querystring, for result consistency we'll ignore it)
(-> (e/get-url driver) (str/split #"\?") first)
;; => "https://en.wikipedia.org/wiki/Clojure"
;; and our new title
(e/get-title driver)
;; => "Clojure - Wikipedia"
;; does the page have Clojure in it?
(e/has-text? driver "Clojure")
;; => true
;; navigate through history
(e/back driver)
(e/forward driver)
(e/refresh driver)
(e/get-title driver)
;; => "Clojure - Wikipedia"
;; let's explore the info box
;; What's its caption? Let's select it with a css query:
(e/get-element-text driver {:css "table.infobox caption"})
;; => "Clojure"
;; Ok, now let's try something trickier
;; Maybe we are interested in what value the infobox holds for the Family row:
(let [wikitable (e/query driver {:css "table.infobox.vevent tbody"})
row-els (e/query-all-from driver wikitable {:tag :tr})]
(for [row row-els
:let [header-col-text (e/with-http-error
(e/get-element-text-el driver
(e/query-from driver row {:tag :th})))]
:when (= "Family" header-col-text)]
(e/get-element-text-el driver (e/query-from driver row {:tag :td}))))
;; => ("Lisp")
;; Etaoin gives you many options; we can do the same-ish in one swoop in XPath:
(e/get-element-text driver "//table[@class='infobox vevent']/tbody/tr/th[text()='Family']/../td")
;; => "Lisp"
;; When we are done, we quit, which stops the Firefox WebDriver
(e/quit driver) ;; the Firefox Window should close
Most api functions require the driver as the first argument.
The doto
macro can give your code a DSL feel.
A portion of the above rewritten with doto
:
(require '[etaoin.api :as e]
'[etaoin.keys :as k])
(def driver (e/firefox))
(doto driver
(e/go "https://en.wikipedia.org/")
(e/set-window-size {:width 1280 :height 800})
(e/wait-visible [{:tag :input :name :search}])
(e/fill {:tag :input :name :search} "Clojure programming language")
(e/wait 1)
(e/fill {:tag :input :name :search} k/enter)
(e/wait-visible {:class :mw-search-results})
(e/click [{:class :mw-search-results} {:class :mw-search-result-heading} {:tag :a}])
(e/wait-visible {:id :firstHeading})
(e/quit))
We encourage you to try the examples from this user guide in your REPL.
The Internet is constantly changing, making testing against live sites impractical. The code in this user guide has been tested to work against our little sample page.
Until we figure out something more clever, it might be easiest to clone the Etaoin GitHub repository and run a REPL from its project root.
Unless otherwise directed, our examples throughout the rest of this guide will assume you’ve already executed the equivalent of:
(require '[etaoin.api :as e]
'[etaoin.keys :as k]
'[clojure.java.io :as io])
(def sample-page (-> "doc/user-guide-sample.html" io/file .toURI str))
(def driver (e/chrome)) ;; or replace chrome with your preference
(e/go driver sample-page)
You can use fill-multi
to shorten the code like so:
(e/fill driver :uname "username")
(e/fill driver :pw "pass")
(e/fill driver :text "some text")
;; let's get what we just set:
(mapv #(e/get-element-value driver %) [:uname :pw :text])
;; => ["username" "pass" "some text"]
into:
;; issue a browser refresh
(e/refresh driver)
(e/fill-multi driver {:uname "username2"
:pw "pass2"
:text "some text2"})
;; to get what we just set:
(mapv #(e/get-element-value driver %) [:uname :pw :text])
;; => ["username2" "pass2" "some text2"]
If any exception occurs during a browser session, the WebDriver process might hang around until you kill it manually.
To prevent that, we recommend the with-<browser>
macros:
(e/with-firefox driver
(doto driver
(e/go "https://google.com")
;; ... your code here
))
This will ensure that the WebDriver process is closed regardless of what happens.
The sections that follow describe how to use Etaoin in more depth.
In addition to these docs, the Etaoin api tests are also a good reference.
Etaoin provides many ways to create a WebDriver instance.
As mentioned, we recommend the with-<browser> convention when you need proper cleanup.
|
Let’s say we want to create a chrome headless driver:
(require '[etaoin.api :as e])
;; at the base we have:
(def driver (e/boot-driver :chrome {:headless true}))
;; do stuff
(e/quit driver)
;; This can also be expressed as:
(def driver (e/chrome {:headless true}))
;; do stuff
(e/quit driver)
;; Or...
(def driver (e/chrome-headless))
;; do stuff
(e/quit driver)
The with-<browser>
functions handle cleanup nicely:
(e/with-chrome {:headless true} driver
(e/go driver "https://clojure.org"))
(e/with-chrome-headless driver
(e/go driver "https://clojure.org"))
Replace chrome
with firefox
, edge
or safari
for other variants.
See API docs for details.
See Driver Options for all options available when creating a driver.
Queries (aka selectors) are used to select the elements on the page that Etaoin will interact with.
;; let's start anew by refreshing the page:
(e/refresh driver)
;; select the element with an html attribute id of 'uname' and fill it with text:
(e/fill driver {:id "uname"} "Etaoin")
;; select the first element with an html button tag and click on it:
(e/click driver {:tag :button})
|
An exception is thrown if a query does not find an element.
Use
|
:active
finds the currently active element.
Note that querying for :active
is deprecated.
Users are encouraged to call get-active-element
to retrieve the active element.
The Google homepage, for example, automatically places the focus on the search input, so there is no need to click on it first.
;; Deprecated
(e/go driver "https://google.com")
(e/fill driver :active "Let's search for something" k/enter)
;; Better
(e/go driver "https://google.com")
(e/fill-el driver (e/get-active-element driver) "Let's search for something" k/enter)
;; Or best... This case is so common that the API includes `fill-active`
(e/go driver "https://google.com")
(e/fill-active driver "Let's search for something" k/enter)
any other keyword is translated to an HTML ID attribute.
Note that you cannot query for an HTML element with an ID of "active" using this keyword syntax because the special keyword :active
conflicts.
If you want to query for an element with an ID of "active", use the map syntax, described below (e.g., {:id "active"}
or {:id :active}
).
(e/go driver sample-page)
(e/fill driver :uname "Etaoin" k/enter)
;; alternatively you can:
(e/fill driver {:id "uname"} "Etaoin Again" k/enter)
a string containing either an XPath or a CSS expression.
The way query
interprets this string depends on the driver’s :locator
setting.
By default, the driver’s :locator
is set to "xpath"
.
The :locator
can be changed using the use-css
, with-css
, use-xpath
, and with-xpath
functions and macros.
(Be careful when writing XPath manually; see Troubleshooting.)
Here, we find an input
tag with an attribute id
of uname
and an attribute name
of username
:
(e/refresh driver)
(e/fill driver ".//input[@id='uname'][@name='username']" "XPath can be tricky")
;; let's check if that worked as expected:
(e/get-element-value driver :uname)
;; => "XPath can be tricky"
;; let's modify the driver to use CSS rather than XPath
;; note that this returns a new, modified copy of the driver
;; the old driver is still valid, however
(def driver-css (e/use-css driver))
(e/refresh driver-css)
(e/fill driver-css "input#uname[name='username']" "CSS can be tricky, too")
(e/get-element-value driver-css :uname)
;; => "CSS can be tricky, too"
a map with either :xpath
or :css
key with a string in corresponding syntax:
(e/refresh driver)
(e/fill driver {:xpath ".//input[@id='uname']"} "XPath selector")
(e/fill driver {:css "input#uname[name='username']"} " CSS selector")
;; And here's what we should see in username input field now:
(e/get-element-value driver :uname)
;; => "XPath selector CSS selector"
This CSS selector reference may be of help.
A query can also be a map that represents an XPath expression as data. The rules are:
A :tag
key represents a tag’s name.
Defaults to *
.
Any non-special key represents an attribute and its value.
:fn/
is a prefix followed by a supported query function.
There are several query functions of the form :fn/*
.
Each query function takes a parameter which is the value associated with the query function keyword in the map.
:fn/index
: Takes a positive integer parameter.
This expands into a trailing XPath [x]
clause and is useful when you need to select a specific row in a table, for example.
:fn/text
: Takes a string parameter. Matches if the element has the exact text specified.
:fn/has-text
: Takes a string parameter.
Matches if the element includes the specified text.
:fn/has-string
: Takes a string parameter.
Matches if the element string contains the specified string.
The difference between :fn/has-text
and :fn/has-string
is the difference between the XPath text()
and string()
functions (text()
is the text within a given element, and string()
is the text of all descendant elements concatenated together in document order).
Generally, if you are targeting an element at the top of the hierarchy, you probably want :fn/has-string
.
If you are targeting a single element at the bottom of the hierarchy, you probably want to use :fn/has-text
.
:fn/has-class
: Takes a string parameter.
Matches if the element’s class
attribute includes the string. Unlike using a :class
key in the map, :fn/has-class
can match single classes, whereas :class
is an exact match of the whole class string.
:fn/has-classes
: Takes a vector of strings parameter.
Matches if the element’s class
attribute includes all of the specified class strings.
:fn/link
: Takes a string parameter.
Matches if the element’s href
attribute contains the specified string.
:fn/enabled
: Takes a boolean (true
or false
) parameter.
If the parameter is true
, matches if the element is enabled.
If the parameter is false
, matches if the element is disabled.
:fn/disabled
: Takes a boolean (true
or false
) parameter.
If the parameter is true
, matches if the element is disabled.
If the parameter is true
, matches if the element is enabled.
Here are some examples of the map syntax:
find the first div
tag
(= (e/query driver {:tag :div})
;; equivalent via xpath:
(e/query driver ".//div"))
;; => true
find the n-th (1-based) div
tag
(= (e/query driver {:tag :div :fn/index 1})
;; equivalent via xpath:
(e/query driver ".//div[1]"))
;; => true
find the tag a
where the class attribute equals to active
(= (e/query driver {:tag :a :class "active"})
;; equivalent xpath:
(e/query driver ".//a[@class='active']"))
find a form by its attributes:
(= (e/query driver {:tag :form :method :GET :class :formy})
;; equivalent in xpath:
(e/query driver ".//form[@method=\"GET\"][@class='formy']"))
find a button by its text (exact match):
(= (e/query driver {:tag :button :fn/text "Submit Form"})
;; equivalent in xpath:
(e/query driver ".//button[text() = 'Submit Form']"))
find an nth element (p
, div
, whatever, it does not matter) with "blarg" text:
(e/get-element-text driver {:fn/has-text "blarg" :fn/index 3})
;; => "blarg in a p"
;; equivalent in xpath:
(e/get-element-text driver ".//*[contains(text(), 'blarg')][3]")
;; => "blarg in a p"
find an element that includes a class:
(e/get-element-text driver {:tag :span :fn/has-class "class1"})
;; => "blarg in a span"
;; equivalent xpath:
(e/get-element-text driver ".//span[contains(@class, 'class1')]")
;; => "blarg in a span"
find an element that has the following domain in a href
:
(e/get-element-text driver {:tag :a :fn/link "clojure.org"})
;; => "link 3 (clojure.org)"
;; equivalent xpath:
(e/get-element-text driver ".//a[contains(@href, \"clojure.org\")]")
;; => "link 3 (clojure.org)"
find an element that includes all of the specified classes:
(e/get-element-text driver {:fn/has-classes [:class2 :class3 :class5]})
;; => "blarg in a div"
;; equivalent in xpath:
(e/get-element-text driver ".//*[contains(@class, 'class2')][contains(@class, 'class3')][contains(@class, 'class5')]")
;; => "blarg in a div"
find explicitly enabled/disabled input widgets:
;; first enabled input
(= (e/query driver {:tag :input :fn/enabled true})
;; equivalent xpath:
(e/query driver ".//input[@enabled=true()]"))
;; => true
;; first disabled input
(= (e/query driver {:tag :input :fn/disabled true})
;; equivalent xpath:
(e/query driver ".//input[@disabled=true()]"))
;; => true
;; return a vector of all disabled inputs
(= (e/query-all driver {:tag :input :fn/disabled true})
;; equivalent xpath:
(e/query-all driver ".//input[@disabled=true()]"))
;; => true
A query can be a vector of any valid query expressions. For vector queries, every expression matches the output from the previous expression.
A simple, somewhat contrived, example:
(e/click driver [{:tag :html} {:tag :body} {:tag :button}])
;; our sample page shows form submits; did it work?
(e/get-element-text driver :submit-count)
;; => "1"
You may combine both XPath and CSS expressions
Reminder: the leading dot in an XPath expression means starting at the current node |
;; under the html tag (using map query syntax),
;; under a div tag with a class that includes some-links (using css query),
;; click on a tag that has
;; a class attribute equal to active (using xpath syntax):
(e/click driver [{:tag :html} {:css "div.some-links"} ".//a[@class='active']"])
;; our sample page shows link clicks, did it work?
(e/get-element-text driver :clicked)
;; => "link 2 (active)"
Sometimes, you may want to interact with the nth element of a query. Maybe you want to click on the second link within:
<ul>
<li class="search-result">
<a href="a">a</a>
</li>
<li class="search-result">
<a href="b">b</a>
</li>
<li class="search-result">
<a href="c">c</a>
</li>
</ul>
You can use the :fn/index
like so:
(e/click driver [{:tag :li :class :search-result :fn/index 2} {:tag :a}])
;; check click tracker from our sample page:
(e/get-element-text driver :clicked)
;; => "b"
or you can use the nth-child trick with the CSS expression like this:
;; start page anew
(e/refresh driver)
(e/click driver {:css "li.search-result:nth-child(2) a"})
(e/get-element-text driver :clicked)
;; => "b"
Finally it is also possible to obtain the nth element directly by using query-all
:
;; start page anew
(e/refresh driver)
(e/click-el driver (nth (e/query-all driver {:css "li.search-result a"}) 1))
(e/get-element-text driver :clicked)
;; => "b"
Notice:
|
query-tree
pipes selectors.
Every selector queries elements from the previous one.
The first selector finds elements from the root, and subsequent selectors find elements downward from each of the previously found elements.
Given the following HTML:
<div id="query-tree-example">
<div id="one">
<a href="#">a1</a>
<a href="#">a2</a>
<a href="#">a3</a>
</div>
<div id="two">
<a href="#">a4</a>
<a href="#">a5</a>
<a href="#">a6</a>
</div>
<div id="three">
<a href="#">a7</a>
<a href="#">a8</a>
<a href="#">a9</a>
</div>
</div>
The following query will find a vector of div
tags then return a set of all a
tags under those div
tags:
(->> (e/query-tree driver :query-tree-example {:tag :div} {:tag :a})
(map #(e/get-element-text-el driver %))
sort)
;; => ("a1" "a2" "a3" "a4" "a5" "a6" "a7" "a8" "a9")
The shadow DOM provides a way to attach another DOM tree to a specified element in the normal DOM and have the internals of that tree hidden from JavaScript and CSS on the same page.
When the browser renders the DOM, the elements from the shadow DOM appear at the location where the tree is rooted in the normal DOM.
This provides a level of encapsulation, allowing "components" in the shadow DOM to be styled differently than the rest of the page and preventing conflicts between the normal page CSS and the component CSS.
The shadow DOM is also hidden from normal Web Driver queries (query
) and thus requires a separate set of API calls to query it. For more details about the shadow DOM, see this article at Mozilla Developer Network (MDN).
There are a few terms that are important to understand when dealing with the Shadow DOM. The "shadow root host" is the element in the standard DOM to which a shadow root is attached as a property. The "shadow root" is the top of the shadow DOM tree rooted at the shadow root host.
The following examples use this HTML fragment in the User Guide sample HTML that has a bit of shadow DOM in it.
<span id="not-in-shadow">I'm not in the shadow DOM</span>
<div id="shadow-root-host">
<template shadowrootmode="open">
<span id="in-shadow">I'm in the shadow DOM</span>
<span id="also-in-shadow">I'm also in the shadow DOM</span>
</template>
</div>
Everything in the template
element is part of the shadow DOM.
The div
with the id
of shadow-root-host
is, as the ID suggests, the shadow root host element.
Given this HTML, you can run a standard query
to find the shadow root host and then use get-element-property-el
to return to the "shadowRoot"
property.
Note that the element IDs returned in the following examples will be unique to the specific Etaoin driver and driver session, and you will not see the same IDs.
(e/query driver {:id "shadow-root-host"})
;; an element ID similar to (but not the same as)
;; "78344155-7a53-46fb-a46e-e864210e501d"
(e/get-element-property-el driver (e/query driver {:id "shadow-root-host"}) "shadowRoot")
;; something similar to
;; {:shadow-6066-11e4-a52e-4f735466cecf "ac5ab914-7f93-427f-a0bf-f7e91098fd37"}
(e/get-element-property driver {:id "shadow-root-host"} "shadowRoot")
;; something similar to
;; {:shadow-6066-11e4-a52e-4f735466cecf "ac5ab914-7f93-427f-a0bf-f7e91098fd37"}
If you go this route, you’re going to have to pick apart the return values. The element-id of the shadow root is the string value of the first map key.
You can get the shadow root element ID more directly using Etaoin’s get-element-shadow-root
API.
The query parameter looks for a matching element in the standard DOM and returns its shadow root property.
(e/get-element-shadow-root driver {:id "shadow-root-host"})
;; something similar to
;; "ac5ab914-7f93-427f-a0bf-f7e91098fd37"
If you already have the shadow root host element, you can return its corresponding shadow root element ID using get-element-shadow-root-el
.
(def host (e/query driver {:id "shadow-root-host"}))
(e/get-element-shadow-root-el driver host)
;; something similar to
;; "ac5ab914-7f93-427f-a0bf-f7e91098fd37"
You can test whether an element is a shadow root host using has-shadow-root?
and has-shadow-root-el?
.
(e/has-shadow-root? driver {:id "shadow-root-host"})
;; => true
(e/has-shadow-root-el? driver host)
;; => true
(e/has-shadow-root? driver {:id "not-in-shadow"})
;; => false
Now that you know how to retrieve the shadow root, you can query elements in the shadow DOM using query-from-shadow-root
, query-all-from-shadow-root
, query-from-shadow-root-el
, and query-all-from-shadow-root-el
.
For query-from-shadow-root
and query-all-from-shadow-root
, the q
parameter specifies a query of the normal DOM to find the shadow root host.
If the host is identified, the shadow-q
parameter is a query that is executed within the shadow DOM rooted at the shadow root host.
The query-from-shadow-root-el
and query-all-from-shadow-root-el
allow you to specify the shadow root host element directly, rather than querying for it.
(def in-shadow (e/query-from-shadow-root driver {:id "shadow-root-host"} {:css "#in-shadow"}))
(e/get-element-text-el driver in-shadow)
;; => "I'm in the shadow DOM"
(->> (e/query-all-from-shadow-root driver {:id "shadow-root-host"} {:css "span"})
(map #(e/get-element-text-el driver %)))
;; => ("I'm in the shadow DOM" "I'm also in the shadow DOM")
(def shadow-root (e/get-element-shadow-root-el driver host))
(e/get-element-text-el driver (e/query-from-shadow-root-el driver shadow-root {:css "#in-shadow"}))
;; => "I'm in the shadow DOM"
(->> (e/query-all-from-shadow-root-el driver shadow-root {:css "span"})
(map #(e/get-element-text-el driver %)))
;; > ("I'm in the shadow DOM" "I'm also in the shadow DOM")
In the previous shadow root queries, you should note that we used CSS selectors for the For more information, see the Web Platforms Test Dashobard. |
To interact with elements found via a query
or query-all
function call you have to pass the query result to either click-el
or fill-el
(note the -el
suffix):
(e/click-el driver (first (e/query-all driver {:tag :a})))
You can collect elements into a vector and arbitrarily interact with them at any time:
(e/refresh driver)
(def elements (e/query-all driver {:tag :input :type :text :fn/disabled false}))
(e/fill-el driver (first elements) "This is a test")
(e/fill-el driver (rand-nth elements) "I like tests!")
Some basic interactions are covered under Selecting Elements, here we go into other types of interactions and more detail.
As of this writing, Chrome, and Edge only support filling inputs with UNICODE in the Basic Multilingual Plane. This includes many characters but not many emojis 😢.
Firefox and Safari seem to support UNICODE more generally 🙂.
(e/with-chrome driver
(e/go driver sample-page)
(e/fill driver :uname "ⱾⱺⱮⱸ ᢹⓂ Ᵽ")
(e/get-element-value driver :uname))
;; => "ⱾⱺⱮⱸ ᢹⓂ Ᵽ"
(e/with-firefox driver
(e/go driver sample-page)
(e/fill driver :uname "ⱾⱺⱮⱸ ᢹⓂ Ᵽ plus 👍🔥🙂")
(e/get-element-value driver :uname))
;; => "ⱾⱺⱮⱸ ᢹⓂ Ᵽ plus 👍🔥🙂"
Real people type slowly and make mistakes.
To emulate these characteristics, you can use the fill-human
function.
The following options are enabled by default:
{:mistake-prob 0.1 ;; a real number from 0.1 to 0.9, the higher the number, the more typos will be made
:pause-max 0.2} ;; max typing delay in seconds
which you can choose to override if you wish:
(e/refresh driver)
(e/fill-human driver :uname "soslowsobad"
{:mistake-prob 0.5
:pause-max 1})
;; or just use default options by omitting them
(e/fill-human driver :uname " typing human defaults")
(e/get-element-value driver :uname)
;; => "soslowsobad typing human defaults"
For multiple inputs, use fill-human-multi
(e/refresh driver)
(e/fill-human-multi driver {:uname "login"
:pw "password"
:text "some text"}
{:mistake-prob 0.1
:pause-max 0.1})
The click
function triggers the left mouse click on an element found by a query term:
(e/click driver {:tag :button})
The click
function uses only the first element found by the query, which sometimes leads to clicking on the wrong items.
To ensure there is one and only one element found, use the click-single
function.
It acts the same but raises an exception when querying the page returns multiple elements:
(e/click-single driver {:tag :button :name "submit"})
Although double-clicking is rarely purposefully employed on web sites, some naive users might think it is the correct way to click on a button or link.
A double-click can be simulated with the double-click
function.
It can be used, for example, to check your handling of disallowing multiple form submissions.
(e/double-click driver {:tag :button :name "submit"})
Functions that move the mouse pointer to a specified element before clicking on it:
(e/left-click-on driver {:tag :a})
(e/middle-click-on driver {:tag :a})
(e/right-click-on driver {:tag :a})
A middle mouse click can open a link in a new background tab. The right-click sometimes is used to imitate a context menu in web applications.
An <option>
from a <select>
can be selected via the click
function.
Given the following HTML:
<select id="dropdown" name="options">
<option value="o1">foo one</option>
<option value="o2">bar two</option>
<option value="o3">bar three</option>
<option value="o4">bar four</option>
</select>
Click on the option with value o4
:
(e/click driver [{:id :dropdown} {:value "o4"}])
(e/get-element-value driver :dropdown)
;; => "o4"
Click on the option with text bar three
:
(e/click driver [{:id :dropdown} {:fn/text "bar three"}])
(e/get-element-value driver :dropdown)
;; => "o3"
Safari Quirk: You might need to first click on the select element, then the option.
|
Etaoin also includes the select convenience function. It will select the first option from a dropdown that includes the specified text. It also automatically handles the Safari quirk. Click the first matching option with text
The same operation, expressed with
|
There is an option to input a series of keys simultaneously. This is useful to imitate holding a system key like Control, Shift, or whatever when typing.
The namespace etaoin.keys
includes key constants as well as a set of functions related to keyboard input.
(require '[etaoin.keys :as k])
A quick example of entering ordinary characters while holding Shift:
(e/refresh driver)
(e/wait 1) ;; maybe we need a sec for active element to focus
(e/fill-active driver (k/with-shift "caps is great"))
(e/get-element-value-el driver (e/get-active-element driver))
;; => "CAPS IS GREAT"
The main input gets populated with "CAPS IS GREAT". Let’s duplicate the text via select-all, copy, and paste keyboard shortcuts:
(if (= "Mac OS X" (System/getProperty "os.name"))
(e/fill-active driver (k/with-command "a") (k/with-command "c") k/arrow-right " " (k/with-command "v"))
(e/fill-active driver (k/with-ctrl "a") (k/with-ctrl "c") k/arrow-right " " (k/with-ctrl "v")))
(e/get-element-value-el driver (e/get-active-element driver))
;; => "CAPS IS GREAT CAPS IS GREAT"
And now let’s clear the input by: 1. moving the cursor to the beginning of the input field with the home key 2. moving the cursor to the end field while holding shift to select all text 3. deleting the selected text with the delete key
(e/fill-active driver k/home (k/with-shift k/end) k/delete)
(e/get-element-value-el driver (e/get-active-element driver))
;; => ""
These functions do not apply to the global browser’s shortcuts. For example, neither "Command + R" nor "Command + T" reload the page or open a new tab. |
The etaoin.keys/with-*
functions are just wrappers for the etaoin.keys/chord
function that might be used for complex cases.
Clicking on a file input button opens an OS-specific dialog.
You technically cannot interact with this dialog using the WebDriver protocol.
Use the upload-file
function to attach a local file to a file input widget.
An exception will be thrown if the local file is not found.
;; open a web page that serves uploaded files
(e/go driver "http://nervgh.github.io/pages/angular-file-upload/examples/simple/")
;; bind element selector to variable; you may also specify an id, class, etc
(def file-input {:tag :input :type :file})
;; upload a file from your system to the first file input
(def my-file "env/test/resources/static/drag-n-drop/images/document.png")
(e/upload-file driver file-input my-file)
;; or pass a native Java File object:
(require '[clojure.java.io :as io])
(def my-file (io/file "env/test/resources/static/drag-n-drop/images/document.png"))
(e/upload-file driver file-input my-file)
When interacting with a remote WebDriver process, you’ll need to avoid the local file existence check by using remote-file
like so:
(e/upload-file driver file-input (e/remote-file "/yes/i/really/do/exist.png"))
The remote file is assumed to exist where the WebDriver is running. The WebDriver will throw an error if it does not exist.
Etaoin includes functions to scroll the web page.
The most important one, scroll-query
jumps the first element found with the query term:
(e/go driver sample-page)
;; scroll to the 5th h2 heading
(e/scroll-query driver {:tag :h2} {:fn/index 5})
;; and back up to the first h1
(e/scroll-query driver {:tag :h1})
To jump to the absolute pixel positions, use scroll
:
(e/scroll driver 100 600)
;; or pass a map with x and y keys
(e/scroll driver {:x 100 :y 600})
To scroll relatively by pixels, use scroll-by
with offset values:
;; scroll right by 100 and down by 300
(e/scroll-by driver 100 300)
;; use map syntax to scroll left by 50 and up by 200
(e/scroll-by driver {:x -50 :y -200})
There are two convenience functions to scroll vertically to the top or bottom of the page:
(e/scroll-bottom driver) ;; you'll see the footer...
(e/scroll-top driver) ;; ...and the header again
The following functions scroll the page in all directions:
(e/scroll driver [0 0]) ;; let's start at the top left
(e/scroll-down driver 200) ;; scrolls down by 200 pixels
(e/scroll-down driver) ;; scrolls down by the default (100) number of pixels
(e/scroll-up driver 200) ;; the same, but scrolls up...
(e/scroll-up driver)
(e/scroll-right driver 200) ;; ... and right
(e/scroll-right driver)
(e/scroll-left driver 200) ;; ...left
(e/scroll-left driver)
All scroll actions are carried out via Javascript. Ensure your browser has it enabled. |
You can only interact with items within an individual frame or iframe by first switching to them.
Say you have an HTML layout like this:
<iframe id="frame1" src="...">
<p id="in-frame1">In frame2 paragraph</p>
<iframe id="frame2" src="...">
<p id="in-frame2">In frame2 paragraph</p>
</iframe>
</iframe>
Let’s explore switching to :frame1
.
(e/go driver sample-page)
;; we start in the main page, we can't see inside frame1:
(e/exists? driver :in-frame1)
;; => false
;; switch context to frame with id of frame1:
(e/switch-frame driver :frame1)
;; now we can interact with elements in frame1:
(e/exists? driver :in-frame1)
;; => true
(e/get-element-text driver :in-frame1)
;; => "In frame1 paragraph"
;; switch back to top frame (the main page)
(e/switch-frame-top driver)
To reach nested frames, you can dig down like so:
;; switch to the first top-level iframe with the main page: frame1
(e/switch-frame-first driver)
;; downward to the first iframe with frame1: frame2
(e/switch-frame-first driver)
(e/get-element-text driver :in-frame2)
;; => "In frame2 paragraph"
;; back up to frame1
(e/switch-frame-parent driver)
;; back up to main page
(e/switch-frame-parent driver)
Use the with-frame
macro to temporarily switch to a target frame, then do some work, returning its last expression while preserving your original frame context.
(e/with-frame driver {:id :frame1}
(e/with-frame driver {:id :frame2}
(e/get-element-text driver :in-frame2)))
;; => "In frame2 paragraph"
Use js-execute
to evaluate a Javascript code in the browser:
(e/js-execute driver "alert('Hello from Etaoin!')")
(e/dismiss-alert driver)
Pass any additional parameters to the script with the arguments
array-like object.
(e/js-execute driver "alert(arguments[2].foo)" 1 false {:foo "hello again!"})
(e/dismiss-alert driver)
We have passed 3 arguments:
1
false
{:foo "hello again!}
which is automatically converted to JSON {"foo": "hello again!"}
The alert then presents the foo
field of the 3rd (index 2) argument, which is "hello again!"
.
To return any data to Clojure, add return
into your script:
(e/js-execute driver "return {foo: arguments[2].foo, bar: [1, 2, 3]}"
;; same args as previous example:
1 false {:foo "hello again!"})
;; => {:bar [1 2 3], :foo "hello again!"}
Notice that the JSON has been automatically converted to edn.
Use js-async
to deal with scripts that rely on async strategies such as setTimeout
.
The WebDriver creates and passes a callback as the last argument to your script.
To indicate that work is complete, you must call this callback.
Example:
(e/js-async
driver
"var args = arguments; // preserve the global args
// WebDriver added the callback as the last arg; we grab it here
var callback = args[args.length-1];
setTimeout(function() {
// We call the WebDriver callback, passing with what we want it to return
// In this case we pass we chose to return 42 from the arg we passed in
callback(args[0].foo.bar.baz);
},
1000);"
{:foo {:bar {:baz 42}}})
;; => 42
If you’d like to override the default script timeout, you can do so for the WebDriver session:
;; optionally save the current value for later restoration
(def orig-script-timeout (e/get-script-timeout driver))
(e/set-script-timeout driver 5) ;; in seconds
;; do some stuff
(e/set-script-timeout driver orig-script-timeout)
or for a block of code via with-script-timeout
:
(e/with-script-timeout driver 30
(e/js-async driver "var callback = arguments[arguments.length-1];
//some long operation here
callback('phew,done!');"))
;; => "phew,done!"
The main difference between a program and a human being is that the first operates very fast.
A computer operates so fast that sometimes a browser cannot render new HTML in time.
After each action, you might consider including a wait-<something>
function that polls a browser until the predicate evaluates to true.
Or just (wait <seconds>)
if you don’t care about optimization.
The with-wait
macro might be helpful when you need to prepend each action with (wait n)
.
For example, the following form:
(e/with-wait 1
(e/refresh driver)
(e/fill driver :uname "my username")
(e/fill driver :text "some text"))
is executed something along the lines of:
(e/wait 1)
(e/refresh driver)
(e/wait 1)
(e/fill driver :uname "my username")
(e/wait 1)
(e/fill driver :text "some text")
and thus returns the result of the last form of the original body.
The (doto-wait n driver & body)
acts like the standard doto
but prepends each form with (wait n)
.
The above example, re-expressed with doto-wait
:
(e/doto-wait 1 driver
(e/refresh)
(e/fill :uname "my username")
(e/fill :text "some text"))
This is effectively the same as:
(doto driver
(e/wait 1)
(e/refresh)
(e/wait 1)
(e/fill :uname "my username")
(e/wait 1)
(e/fill :text "some text"))
In addition to with-wait
and do-wait
there are a number of waiting functions: wait-visible
, wait-has-alert
, wait-predicate
, etc (see the full list in the API docs.
They accept default timeout/interval values that can be redefined using the with-wait-timeout
and with-wait-interval
macros.
They all throw if the wait timeout is exceeded.
(e/with-wait-timeout 15 ;; time in seconds
(doto driver
(e/refresh)
(e/wait-visible {:id :last-section})
(e/click {:tag :a})
(e/wait-has-text :clicked "link 1")))
Wait text:
wait-has-text
waits until an element has text anywhere inside it (including inner HTML).
(e/click driver {:tag :a})
(e/wait-has-text driver :clicked "link 1")
wait-has-text-everywhere
like wait-has-text
but searches for text across the entire page
(e/wait-has-text-everywhere driver "ipsum")
When you navigate to a page, the driver waits until the whole page has been completely loaded. That’s fine in most cases but doesn’t reflect the way human beings interact with the Internet.
Change this default behavior with the :load-strategy
option:
:normal
(the default) wait for full page load (everything, include images, etc)
:none
don’t wait at all
:eager
wait for only DOM content to load
For example, the default :normal
strategy:
(e/with-chrome driver
(e/go driver sample-page)
;; by default you'll hang on this line until the page loads
;; (do-something)
)
Load strategy option of :none
:
(e/with-chrome {:load-strategy :none} driver
(e/go driver sample-page)
;; no pause, no waiting, acts immediately
;; (do-something)
)
The :eager
option only works with Firefox at the moment.
Etaoin supports Webdriver Actions. They are described as "virtual input devices". They act as little device input scripts that can even be run simultaneously.
Here, in raw form, we have an example of two actions. One controls the keyboard, the other controls the pointer (mouse).
;; a keyboard input
{:type "key"
:id "some name"
:actions [{:type "keyDown" :value "a"}
{:type "keyUp" :value "a"}
{:type "pause" :duration 100}]}
;; some pointer input
{:type "pointer"
:id "UUID or some name"
:parameters {:pointerType "mouse"}
:actions [{:type "pointerMove" :origin "pointer" :x 396 :y 323}
;; double click
{:type "pointerDown" :duration 0 :button 0}
{:type "pointerUp" :duration 0 :button 0}
{:type "pointerDown" :duration 0 :button 0}
{:type "pointerUp" :duration 0 :button 0}]}
You can create a map manually and send it to the perform-actions
method:
(def keyboard-input {:type "key"
:id "some name"
:actions [{:type "keyDown" :value "e"}
{:type "keyUp" :value "e"}
{:type "keyDown" :value "t"}
{:type "keyUp" :value "t"}
;; duration is in ms
{:type "pause" :duration 100}]})
;; refresh so that we'll be at the active input field
(e/refresh driver)
;; perform our keyboard input action
(e/perform-actions driver keyboard-input)
Or, you might choose to use Etaoin’s action helpers. First, you create the virtual input device:
(def keyboard (e/make-key-input))
and then fill it with the actions:
(-> keyboard
(e/add-key-down k/shift-left)
(e/add-key-down "a")
(e/add-key-up "a")
(e/add-key-up k/shift-left))
Here’s a slightly larger working annotated example:
;; multiple virtual inputs run simultaneously, so we'll create a little helper to generate n pauses
(defn add-pauses [input n]
(->> (iterate e/add-pause input)
(take (inc n))
last))
(let [username (e/query driver :uname)
submit-button (e/query driver {:tag :button})
mouse (-> (e/make-mouse-input)
;; click on username
(e/add-pointer-click-el
username k/mouse-left)
;; pause 10 clicks to allow keyboard action to enter username
;; (key up and down for each of keypress for etaoin)
(add-pauses 10)
;; click on submit button
(e/add-pointer-click-el
submit-button k/mouse-left))
keyboard (-> (e/make-key-input)
;; pause 2 ticks to allow mouse action to first click on username
;; (move to username element + click on it)
(add-pauses 2)
(e/with-key-down k/shift-left
(e/add-key-press "e"))
(e/add-key-press "t")
(e/add-key-press "a")
(e/add-key-press "o")
(e/add-key-press "i")
(e/add-key-press "n")) ]
(e/perform-actions driver keyboard mouse))
To clear the state of virtual input devices, release all currently pressed keys etc, use the release-actions
method:
(e/release-actions driver)
The screenshot
function dumps the currently visible page to a PNG image file on your disk.
Specify any absolute or relative path.
Specify a string:
(e/screenshot driver "target/etaoin-play/screens1/page.png")
or a File
object:
(require '[clojure.java.io :as io])
(e/screenshot driver (io/file "target/etaoin-play/screens2/test.png"))
With Firefox and Chrome, you can also capture a single element within a page, say a div, an input widget, or whatever. It doesn’t work with other browsers at this time.
(e/screenshot-element driver {:tag :form :class :formy} "target/etaoin-play/screens3/form-element.png")
Use with-screenshots
to take a screenshot to the specified directory after each form is executed in the code block.
The file naming convention is <webdriver-name>-<milliseconds-since-1970>.png
(e/refresh driver)
(e/with-screenshots driver "target/etaoin-play/saved-screenshots"
(e/fill driver :uname "et")
(e/fill driver :uname "ao")
(e/fill driver :uname "in"))
this is equivalent to something along the lines of:
(e/refresh driver)
(e/fill driver :uname "et")
(e/screenshot driver "target/etaoin-play/saved-screenshots/chrome-1.png")
(e/fill driver :uname "ao")
(e/screenshot driver "target/etaoin-play/saved-screenshots/chrome-2.png")
(e/fill driver :uname "in")
(e/screenshot driver "target/etaoin-play/saved-screenshots/chrome-3.png")
Use print-page
to print the current page to a PDF file:
(e/with-firefox-headless driver
(e/go driver sample-page)
(e/print-page driver "target/etaoin-play/printed.pdf"))
See API docs for details.
Sometimes it is useful to go a little deeper.
The Etaoin API exposes an abstraction of the W3C WebDriver protocol. This is normally all you need, but sometimes you’ll want to invoke a WebDriver implementation feature that is not part of the WebDriver protocol.
Etaoin talks to the WebDriver process via its execute
function.
You can use this lower level function to send whatever you like to the WebDriver process.
As a real-world example, Chrome supports taking screenshots with transparent backgrounds.
Here we use Etaoin’s execute
function to ask Chrome to do this:
(e/with-chrome driver
;; navigate to our sample page
(e/go driver sample-page)
;; send the Chrome-specific request for a transparent background
(e/execute {:driver driver
:method :post
:path [:session (:session driver) "chromium" "send_command_and_get_result"]
:data {:cmd "Emulation.setDefaultBackgroundColorOverride"
:params {:color {:r 0 :g 0 :b 0 :a 0}}}})
;; and here we take an element screenshot as per normal
(e/screenshot-element driver
{:tag :form}
(str "target/etaoin-play/saved-screenshots/form.png")))
Function get-logs
returns the browser’s console logs as a vector of maps.
Each map has the following structure:
(e/js-execute driver "console.log('foo')")
(e/get-logs driver)
;; [{:level :info,
;; :message "console-api 2:32 \"foo\"",
;; :source :console-api,
;; :timestamp 1654358994253,
;; :datetime #inst "2022-06-04T16:09:54.253-00:00"}]
;; on the 2nd call, for chrome, we'll find the logs empty
(e/get-logs driver)
;; => []
Currently, logs are available in Chrome only. The message text and the source type will vary by browser vendor. Chrome wipes the logs once they have been read.
You can trace events that come from the DevTools panel. This means that everything you see in the developer console now is available through the Etaoin API. This currently only works for Google Chrome.
To start a driver with devtools support enabled, specify a :dev
map.
(require '[etaoin.api :as e])
(e/with-chrome driver {:dev {}}
;; do some stuff
)
The value must not be a map (not nil
).
When :dev
an empty map, the following defaults are used.
{:perf
{:level :all
:network? true
:page? false
:categories [:devtools.network]
:interval 1000}}
We’ll work with a driver that enables everything:
(require '[etaoin.api :as e])
(def driver (e/chrome {:dev
{:perf
{:level :all
:network? true
:page? true
:interval 1000
:categories [:devtools
:devtools.network
:devtools.timeline]}}}))
Under the hood, Etaoin sets up a special perfLoggingPrefs
dictionary inside the chromeOptions
object.
Now that your browser is accumulating these events, you can read them using a special dev
namespace.
The results will be different when you try this, but here’s what I experienced:
(require '[etaoin.dev :as dev])
(e/go driver "https://google.com")
(def reqs (dev/get-requests driver))
;; reqs is a vector of maps
(count reqs)
;; 23
;; what were the request types?
(frequencies (map :type reqs))
;; {:script 6,
;; :other 2,
;; :xhr 4,
;; :image 5,
;; :stylesheet 1,
;; :ping 3,
;; :document 1,
;; :manifest 1}
;; Interesting, we've got Js requests, images, AJAX and other stuff
;; let's take a peek at the last image:
(last (filter #(= :image (:type %)) reqs))
;; {:state 4,
;; :id "14535.6",
;; :type :image,
;; :xhr? false,
;; :url
;; "https://www.google.com/images/searchbox/desktop_searchbox_sprites318_hr.webp",
;; :with-data? nil,
;; :request
;; {:method :get,
;; :headers
;; {:Referer "https://www.google.com/?gws_rd=ssl",
;; :sec-ch-ua-full-version-list
;; "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"102.0.5005.61\", \"Google Chrome\";v=\"102.0.5005.61\"",
;; :sec-ch-viewport-width "1200",
;; :sec-ch-ua-platform-version "\"10.15.7\"",
;; :sec-ch-ua
;; "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
;; :sec-ch-ua-platform "\"macOS\"",
;; :sec-ch-ua-full-version "\"102.0.5005.61\"",
;; :sec-ch-ua-wow64 "?0",
;; :sec-ch-ua-model "",
;; :sec-ch-ua-bitness "\"64\"",
;; :sec-ch-ua-mobile "?0",
;; :sec-ch-dpr "1",
;; :sec-ch-ua-arch "\"x86\"",
;; :User-Agent
;; "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"}},
;; :response
;; {:status nil,
;; :headers
;; {:date "Sat, 04 Jun 2022 00:11:36 GMT",
;; :x-xss-protection "0",
;; :x-content-type-options "nosniff",
;; :server "sffe",
;; :cross-origin-opener-policy-report-only
;; "same-origin; report-to=\"static-on-bigtable\"",
;; :last-modified "Wed, 22 Apr 2020 22:00:00 GMT",
;; :expires "Sat, 04 Jun 2022 00:11:36 GMT",
;; :cache-control "private, max-age=31536000",
;; :content-length "660",
;; :report-to
;; "{\"group\":\"static-on-bigtable\",\"max_age\":2592000,\"endpoints\":[{\"url\":\"https://csp.withgoogle.com/csp/report-to/static-on-bigtable\"}]}",
;; :alt-svc
;; "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
;; :cross-origin-resource-policy "cross-origin",
;; :content-type "image/webp",
;; :accept-ranges "bytes"},
;; :mime "image/webp",
;; :remote-ip "142.251.41.68"},
;; :done? true}
The details of these responses come from Chrome and are subject to changes to Chrome. |
Since we’re mostly interested in AJAX requests, there is a function get-ajax
that does the same but filters XHR requests:
;; refresh to fill the logs again
(e/go driver "https://google.com")
(e/wait 2) ;; give ajax requests a chance to finish
(last (dev/get-ajax driver))
;; {:state 4,
;; :id "14535.59",
;; :type :xhr,
;; :xhr? true,
;; :url
;; "https://www.google.com/complete/search?q&cp=0&client=gws-wiz&xssi=t&hl=en-CA&authuser=0&psi=OtuaYq-xHNeMtQbkjo6gBg.1654315834852&nolsbt=1&dpr=1",
;; :with-data? nil,
;; :request
;; {:method :get,
;; :headers
;; {:Referer "https://www.google.com/",
;; :sec-ch-ua-full-version-list
;; "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"102.0.5005.61\", \"Google Chrome\";v=\"102.0.5005.61\"",
;; :sec-ch-viewport-width "1200",
;; :sec-ch-ua-platform-version "\"10.15.7\"",
;; :sec-ch-ua
;; "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
;; :sec-ch-ua-platform "\"macOS\"",
;; :sec-ch-ua-full-version "\"102.0.5005.61\"",
;; :sec-ch-ua-wow64 "?0",
;; :sec-ch-ua-model "",
;; :sec-ch-ua-bitness "\"64\"",
;; :sec-ch-ua-mobile "?0",
;; :sec-ch-dpr "1",
;; :sec-ch-ua-arch "\"x86\"",
;; :User-Agent
;; "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"}},
;; :response
;; {:status nil,
;; :headers
;; {:bfcache-opt-in "unload",
;; :date "Sat, 04 Jun 2022 04:10:35 GMT",
;; :content-disposition "attachment; filename=\"f.txt\"",
;; :x-xss-protection "0",
;; :server "gws",
;; :expires "Sat, 04 Jun 2022 04:10:35 GMT",
;; :accept-ch
;; "Sec-CH-Viewport-Width, Sec-CH-Viewport-Height, Sec-CH-DPR, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version, Sec-CH-UA-Arch, Sec-CH-UA-Model, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version-List, Sec-CH-UA-WoW64",
;; :cache-control "private, max-age=3600",
;; :report-to
;; "{\"group\":\"gws\",\"max_age\":2592000,\"endpoints\":[{\"url\":\"https://csp.withgoogle.com/csp/report-to/gws/cdt1\"}]}",
;; :x-frame-options "SAMEORIGIN",
;; :strict-transport-security "max-age=31536000",
;; :content-security-policy
;; "object-src 'none';base-uri 'self';script-src 'nonce-xM7BqmSpeu5Zd6usKOP4JA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/cdt1",
;; :alt-svc
;; "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
;; :content-type "application/json; charset=UTF-8",
;; :cross-origin-opener-policy "same-origin-allow-popups; report-to=\"gws\"",
;; :content-encoding "br"},
;; :mime "application/json",
;; :remote-ip "142.251.41.36"},
;; :done? true};; => nil
A typical pattern of get-ajax
usage is the following.
You’d like to check if a certain request has been fired to the server.
So you press a button, wait for a while, and then read the requests made by your browser.
Having a list of requests, you search for the one you need (e.g. by its URL) and then check its state.
The :state
field has the same semantics of the XMLHttpRequest.readyState
.
It’s an integer from 1 to 4 with the same behavior.
To check if a request has been finished, done, or failed, use these predicates:
;; fill the logs
(e/go driver "https://google.com")
(e/wait 2) ;; give ajax requests a chance to finish
(def reqs (dev/get-ajax driver))
;; you'd search for what you are interested in here
(def req (last reqs))
(dev/request-done? req)
;; => true
(dev/request-failed? req)
;; => nil
(dev/request-success? req)
;; => true
Note that request-done?
doesn’t mean the request has succeeded.
It only means its pipeline has reached a final step.
When you read dev logs, you consume them from an internal buffer that gets flushed.
The second call to get-requests or get-ajax will return an empty list.
|
Perhaps you want to collect these logs.
The function dev/get-performance-logs
returns a list of logs, so you accumulate them in an atom or whatever:
;; setup a collector
(def logs (atom []))
;; make requests
(e/refresh driver)
;; collect as needed
(do (swap! logs concat (dev/get-performance-logs driver))
true)
(count @logs)
;; 136
The logs->requests
and logs->ajax
functions convert already fetched logs into requests.
Unlike get-requests
and get-ajax
, they are pure functions and won’t flush anything.
;; convert our fetched requests from our collector atom
(dev/logs->requests @logs)
(last (dev/logs->requests @logs))
;; {:state 4,
;; :id "14535.162",
;; :type :ping,
;; :xhr? false,
;; :url
;; "https://www.google.com/gen_204?atyp=i&r=1&ei=Zd2aYsrzLozStQbzgbqIBQ&ct=slh&v=t1&m=HV&pv=0.48715273690818806&me=1:1654316389931,V,0,0,1200,1053:0,B,1053:0,N,1,Zd2aYsrzLozStQbzgbqIBQ:0,R,1,1,0,0,1200,1053:93,x:42832,e,U&zx=1654316432856",
;; :with-data? true,
;; :request
;; {:method :post,
;; :headers
;; {:Referer "https://www.google.com/",
;; :sec-ch-ua-full-version-list
;; "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"102.0.5005.61\", \"Google Chrome\";v=\"102.0.5005.61\"",
;; :sec-ch-viewport-width "1200",
;; :sec-ch-ua-platform-version "\"10.15.7\"",
;; :sec-ch-ua
;; "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
;; :sec-ch-ua-platform "\"macOS\"",
;; :sec-ch-ua-full-version "\"102.0.5005.61\"",
;; :sec-ch-ua-wow64 "?0",
;; :sec-ch-ua-model "",
;; :sec-ch-ua-bitness "\"64\"",
;; :sec-ch-ua-mobile "?0",
;; :sec-ch-dpr "1",
;; :sec-ch-ua-arch "\"x86\"",
;; :User-Agent
;; "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"}},
;; :response
;; {:status nil,
;; :headers
;; {:alt-svc
;; "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
;; :bfcache-opt-in "unload",
;; :content-length "0",
;; :content-type "text/html; charset=UTF-8",
;; :date "Sat, 04 Jun 2022 04:20:32 GMT",
;; :server "gws",
;; :x-frame-options "SAMEORIGIN",
;; :x-xss-protection "0"},
;; :mime "text/html",
;; :remote-ip "142.251.41.36"},
;; :done? true}
When working with logs and requests, pay attention to their count and size.
The maps have many keys, and the number of items in collections can be large.
Printing a slew of events might freeze your editor.
Consider using clojure.pprint/pprint
as it relies on max level and length limits.
Sometimes, diagnosing what went wrong during a failed UI test run can be difficult.
Use the with-postmortem
to save useful data to disk before the exception was triggered:
a screenshot of the visible browser page
HTML code of the current browser page
JS console logs, if available for your browser
Example:
(try
(e/with-postmortem driver {:dir "target/etaoin-play/postmortem"}
(e/click driver :non-existing-element))
(catch Exception _e
"yup, we threw!"))
;; => "yup, we threw!"
An exception will occur. Under target/etaoin-postmortem
you will find three postmortem files named like so: <browser>-<host>-<port>-<datetime>.<ext>
, for example:
$ tree target
target
└── etaoin-postmortem
├── chrome-127.0.0.1-49766-2022-06-04-12-26-31.html
├── chrome-127.0.0.1-49766-2022-06-04-12-26-31.json
└── chrome-127.0.0.1-49766-2022-06-04-12-26-31.png
The available with-postmortem
options are:
{;; directory to save artifacts
;; will be created if it does not already exist, defaults to the current working directory
:dir "/home/ivan/UI-tests"
;; directory to save screenshots; defaults to :dir
:dir-img "/home/ivan/UI-tests/screenshots"
;; the same but for HTML sources
:dir-src "/home/ivan/UI-tests/HTML"
;; the same but for console logs
:dir-log "/home/ivan/UI-tests/console"
;; a string template to format a timestamp; See SimpleDateFormat Java class
:date-format "yyyy-MM-dd-HH-mm-ss"}
When creating a WebDriver instance, you can optionally include an options map to tweak the WebDriver and web browser behaviour.
Here, for example, we set an explicit path to the chrome WebDriver binary:
(def driver (e/chrome {:path-driver "/Users/ivan/downloads/chromedriver"}))
:host
for WebDriverDefault: <not set>
Example: :host "192.68.1.12"
When:
Specified, Etaoin attempts to connect to an existing running WebDriver process.
Omitted, Etaoin creates a new local WebDriver process (unless webdriver-url
is specified).
See also :port
, Connecting to an Existing Running WebDriver.
Alternative: see webdriver-url
.
:port
for WebDriverDefault: Etaoin selects a random unused port when launching a local WebDriver process. When connecting to a remote WebDriver process, it varies by vendor:
chrome 9515
firefox 4444
safari 4445
edge 17556
Example: :port 9997
:webdriver-url
Default: <not set>
Example: :web-driver-url "https://chrome.browserless.io/webdriver"
When:
Specified, Etaoin attempts to connect to a pre-existing running WebDriver process.
Omitted, creates a new local WebDriver process (unless :host
for WebDriver is specified).
Alternative: see :host
above.
:path-driver
to WebDriver binaryDefault: Varies by browser vendor:
chrome "chromedriver"
firefox "geckodriver"
safari "safaridriver"
edge "msedgedriver"
Example: :path-driver "/Users/ivan/Downloads/geckodriver"
Typically used if your WebDriver binary is not on your PATH
.
:args-driver
for WebDriver binaryDefault: <not set>
Example: :args-driver ["--binary" "/path/to/firefox/binary"]
(geckodriver specific, you’d probably use :path-browser
to web browser binary instead)
Specifies extra command line arguments to the WebDriver binary.
:webdriver-failed-launch-retries
Default:
:safari
driver 4
all other drivers 0
Example: :webdriver-failed-launch-retries 3
Introduced to compensate for mysterious but recoverable failed launches of safaridriver.
:path-browser
to web browser binaryDefault: <not set>, The WebDriver implementation will make an attempt to find the web browser binary.
Example: :path-browser "/Users/ivan/Downloads/firefox/firefox
Typically used if your web browser binary is not on your PATH
.
:args
for web browser binaryDefault: <not set>
Example: :args ["--incognito" "--app" "http://example.com"]
Specifies extra command line arguments for the web browser binary; see vendor docs for what is available.
:log-level
for the web browser consoleDefault: :all
Example: :log-level :err
Web browser minimal console log level. Only messages with this level and above will be collected. From least to most verbose:
nil
, :off
or :none
for no messages
:err
, :error
, :severe
, :crit
or :critical
:warn
or :warning
:debug
:all
for all messages.
Applies to Chrome and Edge only.
:driver-log-level
Default: <not set>
Example: :driver-log-level "INFO"
WebDriver minimal log level. Values vary by browser driver vendor:
chrome & edge "OFF"
"SEVERE"
"WARNING"
"INFO"
or "DEBUG"
firefox "fatal"
"error"
"warn"
"info"
"config"
"debug"
or "trace"
safari "debug"
- safaridriver
has only one detailed log level which we enable via its --diagnose
option and abstract via "debug"
only logs to a log file which Etaoin automatically discovers and populates as :driver-log-file
in the driver
map
see :post-stop-fns
for one way to dump this log file
:log-stdout
& :log-stderr
for WebDriver outputDefault: no logging: /dev/null
, on Windows NUL
Example:
:log-stdout "target/chromedriver-out.log"
:log-stderr "target/chromedriver-err.log"
Specify :inherit
to have WebDriver process output destination inherit from its calling process (for example, the console or some existing redirection to a file).
:driver-log-file
for discovered WebDriver log fileDefault: <not set>
Etaoin will set this value for you
Example: <n/a>
Not set by user
Only populated for safaridriver when :driver-log-level
set to "debug"
.
:post-stop-fns
to hook in behaviour at driver stopDefault: <not set>
Example:
One usage is to dump safaridriver :driver-log-file
.
:post-stop-fns [(fn dump-discovered-log [driver]
(if-let [log (:driver-log-file driver)]
(do
(println "-[start]-safaridriver log file" log)
(with-open [in (io/input-stream log)]
(io/copy in *out*))
(println "-[end]-safaridriver log file" log))
(println "-no safaridriver log file discovered-")))]
:profile
path to web browser profileDefault: <not set>
Example:
:profile "/Users/ivan/Library/Application Support/Firefox/Profiles/iy4iitbg.Test"
Path to custom web browser profile, see Setting the Browser Profile.
:env
variables for WebDriver processDefault: <not set>
Example: :env {:MOZ_CRASHREPORTER_URL "http://test.com"}
Map of extra environment variables to use when launching the Webdriver process.
:size
of intial web browser windowDefault: [1024 680]
Example: size: [640 480]
Initial web browser window width and height in pixels.
:url
to open in web browserDefault: <not set>
Example: :url "https://clojure.org"
Only works with Firefox at this time.
:user-agent
for web browserDefault: <not set>, Governed by WebDriver/web browser vendor
Example: :user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
Overrides the web browser returned User-Agent
header.
:download-dir
for web browserDefault: <not set>, Governed by web browser vendor
Example: :download-dir "target/chrome-downloads"
The directory where the web browser downloads files.
:headless
web browserDefault: Normally false
, but automatically true
for driver creation functions like chrome-headless
, with-chrome-headless
, etc.
Example: :headless true
Run the web browser without a user interface. See Using Headless Drivers.
:prefs
for web browserDefault: <not set>
Example: see one usage in File Download Directory.
Map of web browser specific preferences.
:load-strategy
Default: :normal
Example: :load-strategy :none
Controls how long the WebDriver should wait before interacting with a page. See Load Strategy.
:capabilities
Default: <not set>
Example: See HTTP Proxy for an example usage.
A WebDriver's capabilities can be vendor-specific and can specify preferred options.
Read WebDriver vendor docs before setting anything here.
While reading docs, note that Etaoin passes along :capabilities
under firstMatch
.
Google Chrome, Firefox, and Microsoft Edge can be run in headless mode. When headless, none of the UI windows appear on the screen. Running without a UI is helpful when:
running integration tests on servers that do not have a graphical output device
running local tests without having them take over your local UI
Ensure your browser supports headless mode by checking if it accepts --headless
command-line argument when running it from the terminal.
When starting a driver, pass the :headless
boolean flag to switch into headless mode.
This flag is ignored for Safari, which, as of August 2024, still does not support headless mode.
(require '[etaoin.api :as e])
(def driver (e/chrome {:headless true})) ;; runs headless Chrome
;; do some stuff
(e/quit driver)
or
(def driver (e/firefox {:headless true})) ;; runs headless Firefox
;; you can also check if a driver is in headless mode:
(e/headless? driver)
;; => true
(e/wait 1) ;; seems to appease Firefox on Linux
(e/quit driver)
There are several shortcuts to run Chrome or Firefox in headless mode:
(def driver (e/chrome-headless))
;; do some stuff
(e/quit driver)
;; or
(def driver (e/firefox-headless {:log-level :all})) ;; with extra settings
;; do some stuff
(e/quit driver)
;; or
(require '[etaoin.api :as e])
(e/with-chrome-headless driver
(e/go driver "https://clojure.org"))
(e/with-firefox-headless {:log-level :all} driver ;; notice extra settings
(e/go driver "https://clojure.org"))
There are also the when-headless
and when-not-headless
macros that conditionally execute a block of commands:
(e/with-chrome driver
(e/when-not-headless driver
;;... some actions that might not be available in headless mode
)
;;... common actions for both versions
)
To specify a directory where the browser should download files, use the :download-dir
option:
(def driver (e/chrome {:download-dir "target/etaoin-play/chrome-downloads"}))
;; do some downloading
(e/driver quit)
Now, when you click on a download link, the file will be saved to that folder. Currently, only Chrome, Edge and Firefox are supported.
Firefox requires specifying MIME-types of the files that should be downloaded without showing a system dialog.
By default, when the :download-dir
parameter is passed, the library adds the most common MIME-types: archives, media files, office documents, etc.
If you need to add your own one, override that Firefox preference manually via the :prefs
option:
(def driver (e/firefox {:download-dir "target/etaoin-play/firefox-downloads"
:prefs {:browser.helperApps.neverAsk.saveToDisk
"some-mime/type-1;other-mime/type-2"}}))
;; do some downloading
(e/driver quit)
To check whether a file was downloaded during UI tests, see Check Whether a File has been Downloaded.
Set a custom User-Agent
header with the :user-agent
option when creating a driver, for example:
(e/with-firefox {:user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"}
driver
(e/wait 1) ;; seems to appease Firefox on Linux
(e/get-user-agent driver))
;; => "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
Setting this header is important when using headless browsers as many websites implement some sort of blocking when the User-Agent includes the "headless" string. This can lead to 403 responses or some weird behavior on the site.
To set proxy settings use environment variables HTTP_PROXY
/HTTPS_PROXY
or pass a map of the following type:
{:proxy {:http "some.proxy.com:8080"
:ftp "some.proxy.com:8080"
:ssl "some.proxy.com:8080"
:socks {:host "myproxy:1080" :version 5}
:bypass ["http://this.url" "http://that.url"]
:pac-url "localhost:8888"}}
;; example
(e/chrome {:proxy {:http "some.proxy.com:8080"
:ssl "some.proxy.com:8080"}})
A :pac-url is for a proxy autoconfiguration file.
Used with Safari as other proxy options do not work in Safari.
|
To fine-tune the proxy you use the original object and pass it as capabilities:
(e/chrome {:capabilities
{:proxy
{:proxyType "manual"
:proxyAutoconfigUrl "some.proxy.com:8080"
:ftpProxy "some.proxy.com:8080"
:httpProxy "some.proxy.com:8080"
:noProxy ["http://this.url" "http://that.url"]
:sslProxy "some.proxy.com:8080"
:socksProxy "some.proxy.com:1080"
:socksVersion 5}}})
To connect to an existing WebDriver, specify the :host
parameter.
When neither the :host nor the :webdriver-url parameter is specified, Etaoin will launch a new WebDriver process.
|
The :host
can be a hostname (localhost, some.remote.host.net) or an IP address (127.0.0.1, 183.102.156.31).
If the port is not specified, the default :port
is assumed.
Both :host
and :port
are ignored if :webdriver-url
is specified.
Example:
;; Connect to an existing chromedriver process on localhost on port 9515
(def driver (e/chrome {:host "127.0.0.1" :port 9515})) ;; for connection to driver on localhost on port 9515
;; Connect to an existing geckodriver process on remote most on default port
(def driver (e/firefox {:host "192.168.1.11"})) ;; the default port for firefox is 4444
;; Connect to a chrome instance on browserless.io via :webdriver-url
;; (replace YOUR-API-TOKEN with a valid browserless.io api token if you want to try this out)
(e/with-chrome {:webdriver-url "https://chrome.browserless.io/webdriver"
:capabilities {"browserless:token" "YOUR-API-TOKEN"
"chromeOptions" {"args" ["--no-sandbox"]}}}
driver
(e/go driver "https://en.wikipedia.org/")
(e/wait-visible driver [{:id :simpleSearch} {:tag :input :name :search}])
(e/fill driver {:tag :input :name :search} "Clojure programming language")
(e/fill driver {:tag :input :name :search} k/enter)
(e/get-title driver))
;; => "Clojure programming language - Search results - Wikipedia"
When running Chrome or Firefox, you may specify a special web browser profile made for test purposes. A profile is a folder that keeps browser settings, history, bookmarks, and other user-specific data.
Imagine, for example, that you’d like to run your integration tests against a user that turned off Javascript execution or image rendering.
This is a hypothetical example. Turning off JavaScript will affect/break certain WebDriver features. And it can affect certain WebDriver implementations, for example. |
In the top right corner of the main window, click on a user button.
In the dropdown, select "Manage People".
Click "Add person", submit a name, and press "Save".
The new browser window should appear. Now, setup the new profile as you want.
Open chrome://version/
page.
Copy the file path that is beneath the Profile Path
caption.
Run Firefox with -P
, -p
, or -ProfileManager
key as the official page describes.
Create a new profile and run the browser.
Setup the profile as needed.
Open about:support
page.
Near the Profile Folder
caption, press the Show in Finder
button.
A new folder window should appear.
Copy its path from there.
Once you’ve got a profile path, launch a driver with the :profile
key as follows:
;; Chrome
(def chrome-profile
"/Users/ivan/Library/Application Support/Google/Chrome/Profile 2/Default")
(def chrome-driver (e/chrome {:profile chrome-profile}))
;; Firefox
(def ff-profile
"/Users/ivan/Library/Application Support/Firefox/Profiles/iy4iitbg.Test")
(def firefox-driver (e/firefox {:profile ff-profile}))
It is not unusual for Continuous Integration services to have no display. This is especially true for Linux runners.
When running your tests on Linux with no display, you have 2 choices:
Run the WebDriver in headless mode
Use a virtual display
Things being what they are, WebDrivers can behave differently when run headless. |
The technologies we use for Etaoin’s CI testing on GitHub Actions for Linux are:
Xvfb - acts as an X virtual display
fluxbox - a lightweight windows manager (needed by geckodriver/Firefox to support window positioning operations)
You can see how we make use of these tools in the Etaoin test script, but in a nutshell:
To install:
sudo apt get install -y xvfb fluxbox
As of this writing Xvfb is pre-installed on the linux runner on GitHub Actions, but fluxbox is not. |
Ensure DISPLAY
env var is set:
export DISPLAY=:99.0
Launch the virtual display and fluxbox:
Xvfb :99 -screen 0 1024x768x24 &
fluxbox -display :99 &
It is desirable to have your tests be independent of one another. One way to achieve this is through the use of a test fixture. The fixture’s job is to, for each test:
create a new driver
run the test with the driver
shutdown the driver
A dynamic *driver*
var might be used to hold the driver.
(ns project.test.integration
"A module for integration tests"
(:require [clojure.test :refer [deftest is use-fixtures]]
[etaoin.api :as e]))
(def ^:dynamic *driver*)
(defn fixture-driver
"Executes a test running a driver. Bounds a driver
with the global *driver* variable."
[f]
(e/with-chrome [driver]
(binding [*driver* driver]
(f))))
(use-fixtures
:each ;; start and stop driver for each test
fixture-driver)
;; now declare your tests
(deftest ^:integration
test-some-case
(doto *driver*
(e/go url-project)
(e/click :some-button)
(e/refresh)
...
))
If, for some reason, you want to reuse a single driver instance for all tests:
(ns project.test.integration
"A module for integration tests"
(:require [clojure.test :refer [deftest is use-fixtures]]
[etaoin.api :as e]))
(def ^:dynamic *driver*)
(defn fixture-browser [f]
(e/with-chrome-headless driver
(e/disconnect-driver driver)
(binding [*driver* driver]
(f))
(e/connect-driver driver)))
;; creating a session every time that automatically erases resources
(defn fixture-clear-browser [f]
(e/connect-driver *driver*)
(e/go *driver* "http://google.com")
(f)
(e/disconnect-driver *driver*))
;; this is run `once` before running the tests
(use-fixtures
:once
fixture-browser)
;; this is run `every` time before each test
(use-fixtures
:each
fixture-clear-browser)
...some tests
For faster testing, you can use this example:
.....
(defn fixture-browser [f]
(e/with-chrome-headless driver
(binding [*driver* driver]
(f))))
;; note that resources, such as cookies, are deleted manually,
;; so this does not guarantee that the tests are clean
(defn fixture-clear-browser [f]
(e/delete-cookies *driver*)
(e/go *driver* "http://google.com")
(f))
......
In the example above, we examined a case when you run tests against a single type of driver. However, you may want to test your site on multiple drivers, say, Chrome and Firefox. In that case, your fixture may become a bit more complex:
(def driver-type [:firefox :chrome])
(defn fixture-drivers [f]
(doseq [type driver-types]
(e/with-driver type {} driver
(binding [*driver* driver]
(testing (format "Testing in %s browser" (name type))
(f))))))
Now, each test will be run twice.
Once for Firefox and then once for Chrome.
Please note the test call is prepended with the testing
macro that puts the driver name into the report.
When a test fails, you’ll know what driver failed the test.
See also Etaoin’s API tests for an example of this strategy. |
To save some artifacts in case of an exception, wrap the body of your test into the with-postmortem
handler as follows:
(deftest test-user-login
(e/with-postmortem *driver* {:dir "/path/to/folder"}
(doto *driver*
(e/go "http://127.0.0.1:8080")
(e/click-visible :login)
;; any other actions...
)))
If any exception occurs in that test, artifacts will be saved.
To avoid copying and pasting the options map, declare it at the top of the module. If you use Circle CI, it would be great to save the data into a special artifacts directory that might be downloaded as a zip file once the build has been finished:
(def pm-dir
(or (System/getenv "CIRCLE_ARTIFACTS") ;; you are on CI
"/some/local/path")) ;; local machine
(def pm-opt
{:dir pm-dir})
Now pass that map everywhere into PM handler:
;; test declaration
(e/with-postmortem *driver* pm-opt
;; test body goes here
)
When an error occurs, you will find a PNG image of your browser page at the time of the exception and an HTML dump.
Since UI tests may take lots of time to pass, it’s definitely a good practice to pass both server and UI tests independently from each other.
If you are using leiningen, here are a few tips.
First, add ^:integration
tag to all the tests that are run under the browser as follows:
(deftest ^:integration
test-password-reset-pipeline
(doto *driver*
(go url-password-reset)
(click :reset-btn)
;; and so on...
))
Then, open your project.clj
file and add test selectors:
:test-selectors {:default (complement :integration)
:integration :integration}
Now, when you launch lein test
you will run all the tests except browser integration tests.
To run integration tests, launch lein test :integration
.
Sometimes, a file downloads automatically when you click a link or visit a page. In tests, you might need to ensure a file has been downloaded successfully. A typical scenario would be:
Provide a custom empty download folder when running a browser (see File Download Directory).
Click on a link or perform any action needed to start the file download.
Wait for some time; for small files, 5-10 seconds would be enough.
Using files API, scan that directory and try to find a new file. Check if it matches a proper extension, name, creation date, etc.
Example:
(require '[clojure.java.io :as io]
'[clojure.string :as str])
;; Local helper that checks whether it is really an Excel file.
(defn xlsx? [file]
(-> file
.getAbsolutePath
(str/ends-with? ".xlsx")))
;; Top-level declarations
(def DL-DIR "/Users/ivan/Desktop")
(def driver (e/chrome {:download-dir DL-DIR}))
;; Later, in tests...
(e/click-visible driver :download-that-application)
(e/wait driver 7) ;; wait for a file has been downloaded
;; Now, scan the directory and try to find a file:
(let [files (file-seq (io/file DL-DIR))
found (some xlsx? files)]
(is found (format "No *.xlsx file found in %s directory." DL-DIR)))
Etaoin can play the files produced by Selenium IDE. Selenium IDE allows you to record web interactions for later playback. It is installed as an optional extension in your web browser.
Once installed and activated, it records your actions to a JSON file with the .side
extension.
You can save that file and run it with Etaoin.
Let’s imagine you’ve installed the IDE and recorded some actions as per Selenium IDE documentation.
Now that you have a test.side
file, you could do something like this:
(require '[clojure.java.io :as io]
'[etaoin.api :as e]
'[etaoin.ide.flow :as flow])
(def driver (e/chrome))
(def ide-file (io/resource "ide/test.side"))
(def opt
{;; The base URL redefines the one from the file.
;; For example, the file was written on the local machine
;; (http://localhost:8080), and we want to perform the scenario
;; on staging (https://preprod-001.company.com)
:base-url "https://preprod-001.company.com"
;; keywords :test-.. and :suite-.. (id, ids, name, names)
;; are used to select specific tests. When not passed,
;; all tests get run. For example:
:test-id "xxxx-xxxx..." ;; a single test by its UUID
:test-name "some-test" ;; a single test by its name
:test-ids ["xxxx-xxxx...", ...] ;; multiple tests by their ids
:test-names ["some-test1", ...] ;; multiple tests by their names
;; the same for suites:
:suite-id ...
:suite-name ...
:suite-ids [...]
:suite-names [...]})
(flow/run-ide-script driver ide-file opt)
Everything related to the IDE feature can be found under the etaoin.ide namespace.
You may also run a .side
script from the command line.
Here is a clojure
example:
clojure -M -m etaoin.ide.main -d firefox -p '{:port 8888}' -r ide/test.side
As well as from an uberjar.
In this case, Etaoin must be in the primary dependencies, not the :dev
or :test
related.
java -cp .../poject.jar -m etaoin.ide.main -d firefox -p '{:port 8888}' -f ide/test.side
We support the following arguments (check them out using the clojure -M -m etaoin.ide.main -h
command):
-d, --driver-name name :chrome The name of driver. The default is `:chrome` -p, --params params {} Parameters for the driver represented as an EDN string, e.g '{:port 8080}' -f, --file path Path to an IDE file on disk -r, --resource path Path to an IDE resource --test-ids ids Comma-separeted test ID(s) --suite-ids ids Comma-separeted suite ID(s) --test-names names Comma-separeted test name(s) --suite-names names Comma-separeted suite name(s) --base-url url Base URL for tests -h, --help
Pay attention to --params
.
This must be an EDN string representing a Clojure map.
That’s the same map that you pass into a driver at creation time.
Please note the IDE support is still experimental. If you encounter unexpected behavior, feel free to open an issue. At the moment, we only support Chrome and Firefox for IDE files.
To work with the WebDrivers in Docker, you can take ready-made images:
Example for Chrome:
docker run --name chromedriver -p 9515:4444 -d -e CHROMEDRIVER_WHITELISTED_IPS='' robcherry/docker-chromedriver:latest
for Firefox:
docker run --name geckodriver -p 4444:4444 -d instrumentisto/geckodriver
To connect to an existing running WebDriver process, you need to specify the :host
.
In this example :host
would be localhost
or 127.0.0.1
.
The :port
would be the appropriate port for the running WebDriver process as exposed by docker.
If the port is not specified, the default port is set.
(def driver (e/chrome-headless {:host "localhost" :port 9515 :args ["--no-sandbox"]}))
(def driver (e/firefox-headless {:host "localhost"})) ;; will try to connect to port 4444
Vendors don’t always do a great job of completely following or implementing the W3C WebDriver Spec.
The web-plaforms-tests dashboard is a great way to get a feel for vendor support of the spec.
Reproduction |
For example,
|
Cause |
This was a bug in |
Solution |
Updating to the current WebDriver resolved the issue. |
Reproduction |
For example, |
Cause |
Likely a bug in chromedriver |
Solution |
Upgrade to newer version that fixes bug. For this particular bug I simply suffered retrying failed tests while waiting for newer version. |
Reproduction |
An attempt to click on a link does nothing. Given the following HTML
An attempt to click on the link does nothing:
|
Cause |
Odd as it may seem, this is a long-standing (maybe on-again-off-again?) bug in |
Work-around |
You can, if you wish, employ JavaScript to click the link:
|
;; we intend to find an element with the text 'some' under an element with id 'mishmash'
(e/get-element-text driver [{:id :mishmash} "//*[contains(text(),'some')]"])
;; => "A little sample page to illustrate some concepts described in the Etaoin user guide."
;; but we've found the first element with text 'some'
In a vector, every expression searches from the previous one in a loop.
Without a leading dot, the XPath "//..."
clause means to find an element from the root of the whole page.
With a dot, it means to find from the current node, which is one from the previous query, and so forth.
Add the XPath dot.
(e/get-element-text driver [{:id :mishmash} ".//*[contains(text(),'some')]"])
;; => "some other paragraph"
;; that's what we were looking for!
(e/click driver :cantseeme)
;; as of this writing, on chrome throws an exception with message containing 'not interactable'
You cannot interact with an element that is not visible or is so small that a human could not click on it.
Selectors for locating elements are not working, even though the elements are clearly available.
Your script may have clicked a link that opened a new tab or window. Even though the new window is in the foreground, the driver instance is still connected to the original window.
Call switch-window-next
when a new tab or window is opened to point the driver to the new tab/window.
when you focus on another window, a WebDriver session that is run under Google Chrome fails.
Google Chrome may suspend a tab when it has been inactive for some time. When the page is suspended, no operation can be done on it. No clicks, Js execution, etc. So try to keep Chrome window active during test session.
When you try to start the driver you get an error:
(def driver (e/firefox {:headless true}))
;; throws an exception containing message with 'invalid argument: can't kill an exited process'
Running Firefox as root in a regular user’s session is not supported
Run the driver with the path to the log files and the "trace" log level and explore the output.
(def driver (firefox {:log-stdout "ffout.log" :log-stderr "fferr.log" :driver-log-level "trace"}))
When you try to start the chromedriver you get an error:
(def driver (e/chrome))
;; throws an exception with message containing 'DevToolsActivePort file doesn't exist'
A common cause for Chrome to crash during startup is running Chrome as root user (administrator) on Linux. While it is possible to work around this issue by passing --no-sandbox flag when creating your WebDriver session, such a configuration is unsupported and discouraged. Ideally you would configure your environment to run Chrome as a regular user instead.
We noticed that the Selenium docker images always invoke chrome with --no-sandbox .
Our test docker images don’t run from the root user, so we’ve not bothered with this.
|
Run driver with argument --no-sandbox
.
Caution!
This bypasses the OS security model.
(e/with-chrome {:args ["--no-sandbox"]} driver
(e/go driver "https://clojure.org"))
A similar problem is described here
Can you improve this documentation? These fine people already did:
Lee Read, lread, Dave Roberts, github-actions[bot], Søren Knudsen, Andrea Tupini, Uday Verma, Roger Erens, Raimon Grau & John KrasnayEdit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close