AndroidEmu: Android Emulation Library Overview
==============================================

Introduction:
-------------

I. Overview:
------------

The Android emulator is now a set of binaries which are built from essentially
two code bases:

- The 'classic' code-base, under $AOSP/external/qemu/, which is based on
  a really old version of QEMU 1.x, heavily patched, implementing a very large
  set of features.

- The 'QEMU2' code-base, under $AOSP/external/qemu-android/, which is based
  on a much more recent version of QEMU 2.x, very lightly patched at the moment,
  but which doesn't implement many Android-specific features to replace the
  classic emulator.

There are other differences, for example:

- The QEMU2 code base supports Android ARM64, MIPS64, x86 and x86_64
  system images (but not 32-bit ARM or MIPS). While the classic one only
  supports 32-bit ARM, MIPS, x86 and x86_64.

- The QEMU2 emulator has much better i/o performance through the use of
  virtio virtual devices.

- The classic emulator has a skinnable UI based on SDL2, with a prototype
  new one based on Qt. The QEMU2 one only displays a mere rectangle.

A small number of Android-specific features have been manually ported to the
QEMU2 code base to enable it to boot Android system images, which meant
rewriting from scratch existing support code (QEMU2 and classic internals are
quite different, so code sharing is actually pretty low).

However, this scheme doesn't really scale, and new features are being added to
the classic emulator at increased speed now, making this manual porting process
even more cumbersome.

To solve this, the classic codebase is going to be refactored into the
following:

  - A static library, named 'AndroidEmu', providing as many Android-specific
    emulation features as possible, in a way that doesn't depend on _any_ QEMU
    internals.

  - An 'engine' (QEMU1) that actually implements the emulation runtime
    and virtual hardware board.

  - A QEMU1-specific 'glue' between these two, that provides the necessary
    plumbing between the above two components.

Meanwhile, the QEMU2 code base will be modified to link against exactly the
same AndroidEmu library, with the help of its own custom 'glue', to form the
final QEMU2 binary.

The goals of AndroidEmu are the following:

  - Being a standalone component that comes with its own set of unit-tests
    that can be run on each build to catch regressions and issues as soon
    as possible (unlike the rest of QEMU1 / QEMU2 which is essentially
    not unit-testable).

  - Providing a UI layer decoupled from the underlying emulation engine
    (be it based on SDL2, as currently, or on Qt, as in the future).

This can be illustrated as follows:

    ____________     ____________     _________        ____________________
   |            |   |            |   |         |      |                    |
   | AndroidEmu | + | QEMU1 Glue | + |  QEMU1  |  ==  |  Classic emulator  |
   |____________|   |____________|   |_________|      |____________________|

    ____________     ____________     _________        ____________________
   |            |   |            |   |         |      |                    |
   | AndroidEmu | + | QEMU2 Glue | + |  QEMU2  |  ==  |   QEMU2 emulator   |
   |____________|   |____________|   |_________|      |____________________|

    ____________     ____________        ______________________________
   |            |   |            |      |                              |
   | AndroidEmu | + | unit tests |  ==  |  AndroidEmu unit-test suite  |
   |____________|   |____________|      |______________________________|


II. AndroidEmu interfaces:
--------------------------

To achieve this, a series of programming techniques will be used, for example
(all paths relative to $AOSP/external/qemu/):

- AndroidEmu provides generic functions that can be called directly from
  glue code. For example:

      android/telephony/: contains generic AndroidEmu code used to support
      GSM and SMS emulation.

      telephony/: contains QEMU1-specific code that uses the generic code
      and plugs it into the rest of the emulation engine (e.g. providing
      a CharDriverState* instance for the emulated modem serial line).

      NOTE: This is the current location, it will be moved to another
            directory that collects all glue code. See sections below.

- AndroidEmu can provide abstract interfaces that are implemented by the
  library in a generic way, but can also be overriden by a QEMU1/QEMU2
  specific implementation. For example (all paths relative to external/qemu/):

    Most of the AndroidEmu code that needs to deal with asynchronous events
    doesn't know if it runs within the QEMU event loop, or a more generic
    one.

    This is done by providing a generic abstract C++ interface to event loops
    (android::base::Looper), which has both a generic implementation, based
    on select(), as well as another one based on the QEMU main event loop
    (android::qemu::Looper).

    The code in AndroidEmu calls android::base::ThreadLooper::get() to
    retrieve the Looper instance for the current thread. By default this
    method lazily creates a generic android::base::Looper on-demand.

    However, when the main QEMU main thread starts, it will create a new
    android::qemu::Looper instance, and inject it through ThreadLooper::set()
    into the current thread.

    As such, any AndroidEmu code that runs in this thread will use QEMU1's
    main event loop without knowing it.

    For more details look at the following sources:

      android/base/async/Looper.h
      android/base/async/Looper.cpp
      android/base/async/ThreadLooper.h
      android-qemu1-glue/base/async/Looper.cpp

    Technically, only the last source file is part of the QEMU1-specific glue.

- The android::base::System class provides an abstract interface to the
  underlying host system. Unit-tests can use an android::base::TestSystem
  instance that will automatically inject itself into the process for the
  duration of the test to ensure that operations are as hermetic as possible.


III. Emulator UI Considerations:
--------------------------------

The code that handles the classic emulator's UI is currently divided into
the following groups :

  ui/console.c:
        QEMU1-specific code to manage console and displays.
        Essentially, QEMU differentiates between 'consoles' and 'displays'.

        A 'console' is an output surface that can hold either text or graphics.
        A given VM may have several consoles enabled at runtime.

        A display, named DisplayState in QEMU1, is something that can "display"
        one console at a time. It is possible to switch which console is
        being associated with a given DisplayState at runtime.

        Finally, a DisplayChangeListener is actually some code that actually
        shows a DisplayState to the screen, or does some specific handling
        (e.g. the VNC DisplayChangeListener can be used to send the
        DisplayState to a remote client using the VNC protocol.

  android-qemu1-glue/display.c:
        The DisplayChangeListener implementation used to send
        the DisplayState to the emulator's UI code.

  android/framebuffer.c:
        helper class acting as a bridge between the virtual framebuffer
        device, and the emulator's DisplayChangeListener.

  android/skin/:
        Main UI emulator code (responsible for displaying the
        emulator's window and its controls, handling use input). This only
        depends on stuff under android/utils/ and android/base/ and thus is
        already completely independent from QEMU1-specific code. However, it
        relies on client code to provide adequate callbacks to operate.

        The skin directory provides an abstract interface as well as an
        implementation backed by the SDL2 library.

  android/skin/qt/:
        An alternative implementation of the abstract skin interface based
        on the Qt widget toolkit (still experimental). Also independent from
        the rest of QEMU1.

        To build it, one must use the --ui=qt flag when calling
        android/configure.sh, android/rebuild.sh or package-release.sh

  android/emulator-window.c:
        The main glue code between the generic skin UI code and the QEMU1
        specific portions of the emulator. Provides appropriate callbacks.

This means that most of the UI code can already be placed in AndroidEmu, and
that it should also be possible to write a fake emulator backend for it in
order to generate a smaller program to test the emulator's UI, as in:

    ______________     __________________        ___________________________
   |              |   |                  |      |                           |
   |  AndroidEmu  | + |  Mock QEMU+Glue  |  ==  |  Android UI test program  |
   |______________|   |__________________|      |___________________________|

The goal of such a program would be to test the UI's features without having
to start a full emulation session (which typically also involves lots of manual
clicks to ensure that everything works correctly at the moment).

This effort is independent from the one described above but should simplify
and speed up work on the UI too.


IV. Refactoring plans:
----------------------

To get where we want, several things are going to change, and this section
tries to details them. Note that all details will probably appear in relevant
entries in http://b.android.com, this is a preview:


IV.1. Repository changes:
- - - - - - - - - - - - -

The manifest files used to manage emulator development branches will be
updated to rename the checkout names of the emulator source directories.

I.e. while the git repositories on android.googlesource.com will be the
same, the following changes in the checkout tree will happen:

  external/qemu          -> external/qemu1
  external/qemu-android  -> external/qemu2

When the source refactoring is completed, we may create a new git repository
named:

  external/android-emu/

To hold the shared AndroidEmu sources, and related build scripts.


IV.2. Source file changes:
- - - - - - - - - - - - - -

A new directory named external/qemu1/android-qemu1-glue/ will be created and
all sources under external/qemu1/android/ that still depend on QEMU-specific
declarations and functions will be moved here first.

The goal here is to keep all 'generic' AndroidEmu code under
external/qemu1/android/ at first (potentially moving it to its own location
like external/android-emu/ in the future).

After this, the code under android-qemu1-glue/ will be refactored into
generic and non-generic parts. The generic ones will be moved to
external/qemu1/android/, the non-generic ones will stay at the same location.

Similarly, external/qemu2/android-qemu2-glue/ will be created, and code
migrated there (however there is currently very little code in QEMU2 that
would be impacted by this).

As a reminder, there is a small amount of QEMU-specific code that deals with
Android that cannot be moved to the glue, due to the way the emulation engine
works. Examples are virtual hardware device implementations, initialization
and teardown code paths, or special features like transparent HTTP proxy
support which requires modifying the 'slirp' network stack (by adding only two
calls to AndroidEmu functions).


IV.3. Build system adjustments:
- - - - - - - - - - - - - - - -

The current build system assumes the following:

  - QEMU2 binaries can be rebuilt as stand-alone binaries before the QEMU1
    ones, through a specialized script (android/scripts/build-qemu-android.sh)

  - 'android/rebuild.sh' will rebuild QEMU1 binaries and copy QEMU2 prebuilts
    to the output directory location.

This only works because QEMU2 sources don't depend on other code from the
project, but this assumption will no longer hold true. As such, the build
system (more precisely android/configure.sh and android/rebuild.sh) will
have to be adjusted to ensure that:

  - AndroidEmu code is built as a standalone static library before anything
    else (with appropriate unit tests, which will be run as part of the
    build routine).

  - QEMU1 and QEMU2 are rebuilt after that, and include AndroidEmu headers,
    and link against the library when generating final executables. For
    example, this requires adding flags like --android-includes=<dir> and
    --android-libdir=<dir> to the QEMU2 'configure' script.

  - Ideally, all of AndroidEmu, QEMU1 and QEMU2 should be rebuilt by
    android/rebuild.sh at the same time (but in the correct order), to
    catch build breaks as soon as possible.

Note that this requires running the QEMU2 'configure' script which is
incredibly slow (e.g. about 15 seconds on a high-end workstation, for each
target host platform). It's probably a good idea to find a way to speed this
process.

Generally speaking, we want to keep the QEMU2 'configure' script as close from
upstream as possible. It's also not possible to put its output into version
control (it creates symlinks and Makefiles that contain hard-coded paths to
the original source directory in the build directory), so a more creative
solution will have to be found. Probably some way to template the 'configure'
output, given that the build uses hermetic (or pseudo-hermetic) toolchains
to build binaries.