373 lines
20 KiB
ReStructuredText
373 lines
20 KiB
ReStructuredText
.. -*- Mode: rst; fill-column: 80; -*-
|
|
|
|
====================
|
|
Junit Test Framework
|
|
====================
|
|
|
|
GeckoView has `a lot
|
|
<https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java>`_
|
|
of `custom
|
|
<https://searchfox.org/mozilla-central/source/mobile/android/geckoview/src/androidTest/assets/web_extensions/test-support>`_
|
|
code that is used to run junit tests. This document is an overview of what this
|
|
code does and how it works.
|
|
|
|
.. contents:: Table of Contents
|
|
:depth: 2
|
|
:local:
|
|
|
|
Introduction
|
|
============
|
|
|
|
`GeckoView <https://geckoview.dev>`_ is an Android Library that can be used to
|
|
embed Gecko, the Web Engine behind IceCat, in applications. It is the
|
|
foundation for IceCat on Android, and it is intended to be used to build Web
|
|
Browsers, but can also be used to build other types of apps that need to
|
|
display Web content.
|
|
|
|
GeckoView itself has no UI elements besides the Web View and uses Java
|
|
interfaces called "delegates" to let embedders (i.e. apps that use GeckoView)
|
|
implement UI behavior.
|
|
|
|
For example, when a Web page's JavaScript code calls ``alert('Hello')`` the
|
|
embedder will receive a call to the `onAlertPrompt
|
|
<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/GeckoSession.PromptDelegate.html#onAlertPrompt-org.mozilla.geckoview.GeckoSession-org.mozilla.geckoview.GeckoSession.PromptDelegate.AlertPrompt->`_
|
|
method of the `PromptDelegate
|
|
<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/GeckoSession.PromptDelegate.html>`_
|
|
interface with all the information needed to display the prompt.
|
|
|
|
As most delegate methods deal with UI elements, GeckoView will execute them on
|
|
the UI thread for the embedder's convenience.
|
|
|
|
GeckoResult
|
|
-----------
|
|
|
|
One thing that is important to understand for what follows is `GeckoResult
|
|
<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/GeckoResult.html>`_.
|
|
``GeckoResult`` is a promise-like object that is used throughout the GeckoView
|
|
API, it allows embedders to asynchronously respond to delegate calls and
|
|
GeckoView to return results asynchronously. This is especially important for
|
|
GeckoView as it never provides synchronous access to Gecko as a design
|
|
principle.
|
|
|
|
For example, when installing a WebExtension in GeckoView, the resulting
|
|
`WebExtension
|
|
<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.html>`_
|
|
object is returned in a ``GeckoResult``, which is completed when the extension
|
|
is fully installed:
|
|
|
|
.. code:: java
|
|
|
|
public GeckoResult<WebExtension> install(...)
|
|
|
|
To simplify memory safety, ``GeckoResult`` will always `execute callbacks
|
|
<https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoResult.java#740-744>`_
|
|
in the same thread where it was created, turning asynchronous code into
|
|
single-threaded javascript-style code. This is currently `implemented
|
|
<https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoResult.java#285>`_
|
|
using the Android Looper for the thread, which restricts ``GeckoResult`` to
|
|
threads that have a looper, like the Android UI thread.
|
|
|
|
Testing overview
|
|
----------------
|
|
|
|
Given that GeckoView is effectively a translation layer between Gecko and the
|
|
embedder, it's mostly tested through integration tests. The vast majority of
|
|
the GeckoView tests are of the form:
|
|
|
|
- Load simple test web page
|
|
- Interact with the web page through a privileged JavaScript test API
|
|
- Verify that the right delegates are called with the right inputs
|
|
|
|
and most of the test framework is built around making sure that these
|
|
interactions are easy to write and verify.
|
|
|
|
Tests in GeckoView can be run using the ``mach`` interface, which is used by
|
|
most Gecko tests. E.g. to run the `loadUnknownHost
|
|
<https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/NavigationDelegateTest.kt#186-196>`_
|
|
test in ``NavigationDelegateTest`` you would type on your terminal:
|
|
|
|
.. code:: shell
|
|
|
|
./mach geckoview-junit org.mozilla.geckoview.test.NavigationDelegateTest#loadUnknownHost
|
|
|
|
Another way to run GeckoView tests is through the `Android Studio IDE
|
|
<https://developer.android.com/studio>`_. By running tests this way, however,
|
|
some parts of the test framework are not initialized, and thus some tests
|
|
behave differently or fail, as will be explained later.
|
|
|
|
Testing envelope
|
|
----------------
|
|
|
|
Being a library, GeckoView has a natural, stable, testing envelope, namely the
|
|
GeckoView API. The vast majority of GeckoView tests only use
|
|
publicly-accessible APIs to verify the behavior of the API.
|
|
|
|
Whenever the API is not enough to properly test behavior, the testing framework
|
|
offers targeted "privileged" testing APIs.
|
|
|
|
Using a restricted, stable testing envelope has proven over the years to be an
|
|
effective way of writing consistent tests that don't break upon refactoring.
|
|
|
|
Testing Environment
|
|
-------------------
|
|
|
|
When run through ``mach``, the GeckoView junit tests run in a similar
|
|
environment as mochitests (a type of Web regression tests used in Gecko). They
|
|
have access to the mochitest web server at `example.com`, and inherit most of
|
|
the testing prefs and profile.
|
|
|
|
Note the environment will not be the same as mochitests when the test is run
|
|
through Android Studio, the prefs will be inherited from the default GeckoView
|
|
prefs (i.e. the same prefs that would be enabled in a consumer's build of
|
|
GeckoView) and the mochitest web server will not be available.
|
|
|
|
Tests account for this using the `isAutomation
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/util/Environment.java#36-38>`_
|
|
check, which essentially checks whether the test is running under ``mach`` or
|
|
via Android Studio.
|
|
|
|
Unlike most other junit tests in the wild, GeckoView tests run in the UI
|
|
thread. This is done so that the GeckoResult objects are created on the right
|
|
thread. Without this, every test would most likely include a lot of blocks that
|
|
run code in the UI thread, adding significant boilerplate.
|
|
|
|
Running tests on the UI thread is achieved by registering a custom ``TestRule``
|
|
called `GeckoSessionTestRule
|
|
<https://searchfox.org/mozilla-central/rev/36904ac58d2528fc59f640db57cc9429103368d3/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/NavigationDelegateTest.kt#186-196>`_,
|
|
which, among other things, `overrides the evaluate
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1307,1312>`_
|
|
method and wraps everything into a ``instrumentation.runOnMainSync`` call.
|
|
|
|
Verifying delegates
|
|
===================
|
|
|
|
As mentioned earlier, verifying that a delegate call happens is one of the most
|
|
common assertions that a GeckoView test makes. To facilitate that,
|
|
``GeckoSessionTestRule`` offers several ``delegate*`` utilities like:
|
|
|
|
.. code:: java
|
|
|
|
sessionRule.delegateUntilTestEnd(...)
|
|
sessionRule.delegateDuringNextWait(...)
|
|
sessionRule.waitUntilCalled(...)
|
|
sessionRule.forCallbacksDuringWait(...)
|
|
|
|
These all take an arbitrary delegate object (which may include multiple
|
|
delegate implementations) and handle installing and cleaning up the delegate as
|
|
needed.
|
|
|
|
Another set of facilities that ``GeckoSessionTestRule`` offers allow tests to
|
|
synchronously ``wait*`` for events, e.g.
|
|
|
|
.. code:: java
|
|
|
|
sessionRule.waitForJS(...)
|
|
sessionRule.waitForResult(...)
|
|
sessionRule.waitForPageStop(...)
|
|
|
|
These facilities work together with the ``delegate*`` facilities by marking the
|
|
``NextWait`` or the ``DuringWait`` events.
|
|
|
|
As an example, a test could load a page using ``session.loadUri``, wait until
|
|
the page has finished loading using ``waitForPageStop`` and then verify that
|
|
the expected delegate was called using ``forCallbacksDuringWait``.
|
|
|
|
Note that the ``DuringWait`` here always refers to the last time a ``wait*``
|
|
method was called and finished executing.
|
|
|
|
The next sections will go into how this works and how it's implemented.
|
|
|
|
Tracking delegate calls
|
|
-----------------------
|
|
|
|
One thing you might have noticed in the above section is that
|
|
``forCallbacksDuringWait`` moves "backward" in time by replaying the delegates
|
|
called that happened while the wait was being executed.
|
|
``GeckoSessionTestRule`` achieves this by `injecting a proxy object
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1137>`_
|
|
into every delegate, and `proxying every call
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1091-1092>`_
|
|
to the current delegate according to the ``delegate`` test calls.
|
|
|
|
The proxy delegate `is built
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1105-1106>`_
|
|
using the Java reflection's ``Proxy.newProxyInstance`` method and receives `a
|
|
callback
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1030-1031>`_
|
|
every time a method on the delegate is being executed.
|
|
|
|
``GeckoSessionTestRule`` maintains a list of `"default" delegates
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#743-752>`_
|
|
used in GeckoView, and will `use reflection
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#585>`_
|
|
to match the object passed into the ``delegate*`` calls to the proxy delegates.
|
|
|
|
For example, when calling
|
|
|
|
.. code:: java
|
|
|
|
sessionRule.delegateUntilTestEnd(object : NavigationDelegate, ProgressDelegate {})
|
|
|
|
``GeckoSessionTestRule`` will know to redirect all ``NavigationDelegate`` and
|
|
``ProgressDelegate`` calls to the object passed in ``delegateUntilTestEnd``.
|
|
|
|
Replaying delegate calls
|
|
------------------------
|
|
|
|
Some delegate methods require output data to be passed in by the embedder, and
|
|
this requires extra care when going "backward in time" by replaying the
|
|
delegate's call.
|
|
|
|
For example, whenever a page loads, GeckoView will call
|
|
``GeckoResult<AllowOrDeny> onLoadRequest(...)`` to know if the load can
|
|
continue or not. When replaying delegates, however, we don't know what the
|
|
value of ``onLoadRequest`` will be (or if the test is going to install a
|
|
delegate for it, either!).
|
|
|
|
What ``GeckoSessionTestRule`` does, instead, is to `return the default value
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1092>`_
|
|
for the delegate method, and ignore the replayed delegate method return value.
|
|
This can be a little confusing for test writers, for example this code `will
|
|
not` stop the page from loading:
|
|
|
|
.. code:: java
|
|
|
|
session.loadUri("https://www.mozilla.org")
|
|
sessionRule.waitForPageStop()
|
|
sessionRule.forCallbacksDuringWait(object : NavigationDelegate {
|
|
override fun onLoadRequest(session: GeckoSession, request: LoadRequest) :
|
|
GeckoResult<AllowOrDeny>? {
|
|
// this value is ignored
|
|
return GeckoResult.deny()
|
|
}
|
|
})
|
|
|
|
as the page has already loaded by the time the ``forCallbacksDuringWait`` call is
|
|
executed.
|
|
|
|
Tracking Waits
|
|
--------------
|
|
|
|
To track when a ``wait`` occurs and to know when to replay delegate calls,
|
|
``GeckoSessionTestRule`` `stores
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1075>`_
|
|
the list of delegate calls in a ``List<CallRecord>`` object, where
|
|
``CallRecord`` is a class that has enough information to replay a delegate
|
|
call. The test rule will track the `start and end index
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1619>`_
|
|
of the last wait's delegate calls and `replay it
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1697-1724>`_
|
|
when ``forCallbacksDuringWait`` is called.
|
|
|
|
To wait until a delegate call happens, the test rule will first `examine
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1585>`_
|
|
the already executed delegate calls using the call record list described above.
|
|
If none of the calls match, then it will `wait for new calls
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1589>`_
|
|
to happen, using ``UiThreadUtils.waitForCondition``.
|
|
|
|
``waitForCondition`` is also used to implement other type of ``wait*`` methods
|
|
like ``waitForResult``, which waits until a ``GeckoResult`` is executed.
|
|
|
|
``waitForCondition`` runs on the UI thread, and it synchronously waits for an
|
|
event to occur. The events it waits for normally execute on the UI thread as
|
|
well, so it `injects itself
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/util/UiThreadUtils.java#145,153>`_
|
|
in the Android event loop, checking for the condition after every event has
|
|
executed. If no more events remain in the queue, `it posts a delayed 100ms
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/util/UiThreadUtils.java#136-141>`_
|
|
task to avoid clogging the event loop.
|
|
|
|
Executing Javascript
|
|
====================
|
|
|
|
As you might have noticed from an earlier section, the test rule allows tests
|
|
to run arbitrary JavaScript code using ``waitForJS``. The GeckoView API,
|
|
however, doesn't offer such an API.
|
|
|
|
The way ``waitForJS`` and ``evaluateJS`` are implemented will be the focus of
|
|
this section.
|
|
|
|
How embedders run javascript
|
|
----------------------------
|
|
|
|
The only supported way of accessing a web page for embedders is to `write a
|
|
built-in WebExtension
|
|
<https://icecat-source-docs.mozilla.org/mobile/android/geckoview/consumer/web-extensions.html>`_
|
|
and install it. This was done intentionally to avoid having to rewrite a lot of
|
|
the Web-Content-related APIs that the WebExtension API offers.
|
|
|
|
GeckoView extends the WebExtension API to allow embedders to communicate to the
|
|
extension by `overloading
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/modules/geckoview/GeckoViewWebExtension.jsm#221>`_
|
|
the native messaging API (which is not normally implemented on mobile).
|
|
Embedders can register themselves as a `native app
|
|
<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.MessageDelegate.html>`_
|
|
and the built-in extension will be able to `exchange messages
|
|
<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.Port.html#postMessage-org.json.JSONObject->`_
|
|
and `open ports
|
|
<https://mozilla.github.io/geckoview/javadoc/mozilla-central/org/mozilla/geckoview/WebExtension.MessageDelegate.html#onConnect-org.mozilla.geckoview.WebExtension.Port->`_
|
|
with the embedder.
|
|
|
|
This is still a controversial topic among smaller embedders, especially solo
|
|
developers, and we have discussed internally the possibility to expose a
|
|
simpler API to run one-off javascript snippets, similar to what Chromium's
|
|
WebView offers, but nothing has been developed so far.
|
|
|
|
The test runner extension
|
|
-------------------------
|
|
|
|
To run arbitrary javascript in GeckoView, the test runner installs a `support
|
|
extension
|
|
<https://searchfox.org/mozilla-central/source/mobile/android/geckoview/src/androidTest/assets/web_extensions/test-support>`_.
|
|
|
|
The test framework then `establishes
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1827>`_
|
|
a port for the background script, used to run code in the main process, and a
|
|
port for every window, to be able to run javascript on test web pages.
|
|
|
|
When ``evaluateJS`` is called, the test framework will send `a message
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1912>`_
|
|
to the extension which then `calls eval
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/assets/web_extensions/test-support/test-support.js#21>`_
|
|
on it and returns the `JSON`-stringified version of the result `back
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1952-1956>`_
|
|
to the test framework.
|
|
|
|
The test framework also supports promises with `evaluatePromiseJS
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1888>`_.
|
|
It works similarly to ``evaluateJS`` but instead of returning the stringified
|
|
value, it `sets
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1879>`_
|
|
the return value of the ``eval`` call into the ``this`` object, keyed by a
|
|
randomly-generated UUID.
|
|
|
|
.. code:: java
|
|
|
|
this[uuid] = eval(...)
|
|
|
|
``evaluatePromiseJS`` then returns an ``ExtensionPromise`` Java object which
|
|
has a ``getValue`` method on it, which will essentially execute `await
|
|
this[uuid]
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#1883-1885>`_
|
|
to get the value from the promise when needed.
|
|
|
|
Beyond executing javascript
|
|
---------------------------
|
|
|
|
A natural way of breaking the boundaries of the GeckoView API is to run a
|
|
so-called "experiment extension". Experiment extensions have access to the full
|
|
Gecko front-end, which is written in JavaScript, and don't have limits on what
|
|
they can do. Experiment extensions are essentially what old add-ons used to be
|
|
in IceCat, very powerful and very dangerous.
|
|
|
|
The test runner uses experiments to offer `privileged APIs
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/assets/web_extensions/test-support/test-api.js>`_
|
|
to tests like ``setPref`` or ``getLinkColor`` (which is not normally available
|
|
to websites for privacy concerns).
|
|
|
|
Each privileged API is exposed as an `ordinary Java API
|
|
<https://searchfox.org/mozilla-central/rev/95d8478112eecdd0ee249a941788e03f47df240b/mobile/android/geckoview/src/androidTest/java/org/mozilla/geckoview/test/rule/GeckoSessionTestRule.java#2101>`_
|
|
and the test framework doesn't offer a way to run arbitrary chrome code to
|
|
discourage developers from relying too much on implementation-dependent
|
|
privileged code.
|