ded133c446fa9d0d23e6bde19a66fb2ce3980491 |
|
31-Jan-2015 |
Svetoslav <svetoslavganov@google.com> |
Fix broken activation of the selected view in accessibility mode. We were using an approximation to determine where to send a pair of down and up events to click on the view that has accessibility focus. We were doing reverse computation to figuring out which portion of the view is not covered by interactive views and get a point in this region. However, determining whether a view is interactive is not feasible in general since for example may override onTouchEvent. This results in views not being activated or which is worse wrong views being activated. This change swithes to a new approach to activate views in accessibility mode which is guaranteed to always work except the very rare case of a view that overrides dispatchTouchEvent (which developers shouldn't be doing). The new approach is to flag the down and up events pair sent by the touch explorer as targeting the accessibility focused view. Such events are dispatched such that views predecessors of the accessibility focus do not handle them guaranteeing that these events reach the accessibiliy focused view. Once the accessibiliy focused view gets such an event it clears the flag and the event is dispatched following the normal event dispatch semantics. The new approach is semantically equivalent to requesting the view to perform a click accessiblitiy action but is more generic as it is not affected by views not implementing click action support correctly. bug:18986806 bug:18889611 Change-Id: Id4b7b886c9fd34f7eb11e606636d8e3bab122869
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
7498efdc5e163d6b4a11db941c7d13c169d37284 |
|
04-Sep-2014 |
Svet Ganov <svetoslavganov@google.com> |
Clicking on partially coverd views by other views or windows. In touch exploration mode an accessibility service can move accessibility focus in response to user gestures. In this case when the user double-taps the system is sending down and up events at the center of the acessibility focused view. This works fine until the clicked view's center is covered by another clickable view. In such a scenario the user thinks he is clicking on one view but the click is handled by another. Terrible. This change solves the problem of clicking on the wrong view and also solves the problem of clicking on the wrong window. The key idea is that when the system detects a double tap or a double tap and hold it asks the accessibility focused node (if such) to compute a point at which a click can be performed. In respinse to that the node is asking the source view to compute this. If a view is partially covered by siblings or siblings of predecessors that are clickable, the click point will be properly computed to ensure the click occurs on the desired view. The click point is also bounded in the interactive part of the host window. The current approach has rare edge cases that may produce false positives or false negatives. For example, a portion of the view may be covered by an interactive descendant of a predecessor, which we do not compute (we check only siblings of predecessors). Also a view may be handling raw touch events instead of registering click listeners, which we cannot compute. Despite these limitations this approach will work most of the time and it is a huge improvement over just blindly sending the down and up events in the center of the view. Note that the additional computational complexity is incurred only when the user wants to click on the accessibility focused view which is very a rare event and this is a good tradeoff. bug:15696993 Change-Id: I85927a77d6c24f7550b0d5f9f762722a8230830f
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
9ae9ed24aa252acbc2e6288068b991dd5291967b |
|
03-Sep-2014 |
Svetoslav <svetoslavganov@google.com> |
Fix AccessibilityNode's isVisibleToUser behavior. The isVisibleToUser property of an AccessibilityNodeInfo specifies whether the user can see the source view. It is used by accessibility services to figure out whether to focus on a view. This property was giving a wrong value if the view is covered by another window such as the keyboard. As a result the user hears one thing but when double taps interacts with the overlaid window which is another thing. bug:15938254 Change-Id: Ib9feb20ea422a24a512c47ed1234961ae0386a7f
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
80943d8daa6ab31ab5c486d57aea406aa0730d58 |
|
02-Jan-2013 |
Svetoslav Ganov <svetoslavganov@google.com> |
Adding UI test automation APIs. This change adds APIs support for implementing UI tests. Such tests do not rely on internal application structure and can span across application boundaries. UI automation APIs are encapsulated in the UiAutomation object that is provided by an Instrumentation object. It is initialized by the system and can be used for both introspecting the screen and performing interactions simulating a user. UI test are normal instrumentation tests and are executed on the device. UiAutomation uses the accessibility APIs to introspect the screen and a special delegate object to perform privileged operations such as injecting input events. Since instrumentation tests are invoked by a shell command, the shell program launching the tests creates a delegate object and passes it as an argument to started instrumentation. This delegate allows the APK that runs the tests to access some privileged operations protected by a signature level permissions which are explicitly granted to the shell user. The UiAutomation object also supports running tests in the legacy way where the tests are run as a Java shell program. This enables existing UiAutomator tests to keep working while the new ones should be implemented using the new APIs. The UiAutomation object exposes lower level APIs which allow simulation of arbitrary user interactions and writing complete UI test cases. Clients, such as UiAutomator, are encouraged to implement higher- level APIs which minimize development effort and can be used as a helper library by the test developer. The benefit of this change is decoupling UiAutomator from the system since the former was calling hidden APIs which required that it is bundled in the system image. This prevented UiAutomator from being evolved separately from the system. Also UiAutomator was creating additional API surface in the system image. Another benefit of the new design is that now test cases have access to a context and can use public platform APIs in addition to the UiAutomator ones. Further, third-parties can develop their own higher level test APIs on top of the lower level ones exposes by UiAutomation. bug:8028258 Also this change adds the fully qualified resource name of the view's id in the emitted AccessibilityNodeInfo if a special flag is set while configuring the accessibility service. Also added is API for looking up node infos by this id. The id resource name is relatively more stable compared to the generaed id number which may change from one build to another. This API facilitate reuing the already defined ids for UI automation. bug:7678973 Change-Id: I589ad14790320dec8a33095953926c2a2dd0228b
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
152e9bb81aa5b2ab4637f4b2dae04b3ce89fa891 |
|
13-Oct-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Refactoring of the screen magnification feature. 1. The screen magnification feature was implemented entirely as a part of the accessibility manager. To achieve that the window manager had to implement a bunch of hooks for an external client to observe its internal state. This was problematic since it dilutes the window manager interface and allows code that is deeply coupled with the window manager to reside outside of it. Also the observer callbacks were IPCs which cannot be called with the window manager's lock held. To avoid that the window manager had to post messages requesting notification of interested parties which makes the code consuming the callbacks to run asynchronously of the window manager. This causes timing issues and adds unnecessary complexity. Now the magnification logic is split in two halves. The first half that is responsible to track the magnified portion of the screen and serve as a policy which windows can be magnified and it is a part of the window manager. This part exposes higher level APIs allowing interested parties with the right permissions to control the magnification of a given display. The APIs also allow a client to be registered for callbacks on interesting changes such as resize of the magnified region, etc. This part servers as a mediator between magnification controllers and the window manager. The second half is a controller that is responsible to drive the magnification state based on touch interactions. It also presents a highlight when magnified to suggest the magnified potion of the screen. The controller is responsible for auto zooming out in case the user context changes - rotation, new actitivity. The controller also auto pans if a dialog appears and it does not interesect the magnified frame. bug:7410464 2. By design screen magnification and touch exploration work separately and together. If magnification is enabled the user sees a larger version of the widgets and a sub section of the screen content. Accessibility services use the introspection APIs to "see" what is on the screen so they can speak it, navigate to the next item in response to a gesture, etc. Hence, the information returned to accessibility services has to reflect what a sighted user would see on the screen. Therefore, if the screen is magnified we need to adjust the bounds and position of the infos describing views in a magnified window such that the info bounds are equivalent to what the user sees. To improve performance we keep accessibility node info caches in the client process. However, when magnification state changes we have to clear these caches since the bounds of the cached infos no longer reflect the screen content which just got smaller or larger. This patch propagates not only the window scale as before but also the X/Y pan and the bounds of the magnified portion of the screen to the introspected app. This information is used to adjust the bounds of the node infos coming from this window such that the reported bounds are the same as the user sees not as the app thinks they are. Note that if magnification is enabled we zoom the content and pan it along the X and Y axis. Also recomputed is the isVisibleToUser property of the reported info since in a magnified state the user sees a subset of the window content and the views not in the magnified viewport should be reported as not visible to the user. bug:7344059 Change-Id: I6f7832c7a6a65c5368b390eb1f1518d0c7afd7d2
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
c9c9a48e7bafae63cb35a9aa69255e80aba83988 |
|
16-Jul-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Removing a workaround for incorrect window position on window move. 1. The window manager was not notifying a window when the latter has been moved. This was causing incorrect coordinates of the nodes reported to accessibility services. To workaround that we have carried the correct window location when making a call from the accessibility layer into a window. Now the window manager notifies the window when it is moved and the workaround is no longer needed. This change takes it out. 2. The left and right in the attach info were not updated properly after a report that the window has moved. 3. The accessibility manager service was calling directly methods on the window manager service without going through the interface of the latter. This leads to unnecessary coupling and in the long rung increases system complexity and reduces maintability. bug:6623031 Change-Id: Iacb734b1bf337a47fad02c827ece45bb2f53a79d
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
86783474fdec98a22bc22e224462767eab13e273 |
|
07-Jun-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Cannot interact with dialogs when IME is up and on not touch explored popups. 1. If the last touch explored location is within the active window we used to click on exact location if it is within the accessibility focus otherwise in the accessibility focus center. If the last touch explored location is not within the active window we used to just click there. This breaks in the case were one has touch explored at a given place in the current window and now a dialog opens *not* covering the touch explored location. If one uses swipes to move accessibility focus i.e. to traverse the dialog without touching it one cannot activate anything because the touch explorer is using the last touch explored location that is outside of the active window e.g the dialog. The solution is to clear the last touch explored location when a window opens or accessibility focus moves. If the last touch explored location is null we are clicking in the accessibility focus location. bug:6620911 2. There is a bug in the window manager that does not notify a window that its location has changed (bug:6623031). This breaks accessibility interaction with dialogs that have input because when the IME is up the dialog is moved but not notified. Now the accessibility layer gets incorrect location for the accessibility focus and the window bounds. The soluion is when the accessibility manager service calls into the remove thress to obtain some accessibility node infos it passes the window left and top which it gets from the window manager. These values are used to update the attach info window left and top so all accessibility node infos emitted from that window had correct bounds in screen coordinates. bug:6620796 Change-Id: I18914f2095c55cfc826acf5277bd94b776bda0c8
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
aa780c110922148a6a4ba06734bb2b0bb8c98f93 |
|
20-Apr-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Adding support for traversing the content of a node info at granularity. 1. A view that creates an accessibility node info may add to the info a list of granularity labels. These are granularities by which the source view can iterate over its content. For example a text view may support character, word link while a web view may additionally support buttons, tables, etc. There are actions on accessibility node info to go to the next/previous at a given granularity which is passesed as an argument. 2. Added Bundle argument to the APIs for performing accessibility actions. This is generic and extensible. bug:5932640 Change-Id: I328cbbb4cddfdee082ab2a8b7ff1bd7477d8d6f9
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
4213804541a8b05cd0587b138a2fd9a3b7fd9350 |
|
20-Mar-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Accessibility focus - framework Usefulness: Keep track of the current user location in the screen when traversing the it. Enabling structural and directional navigation over all elements on the screen. This enables blind users that know the application layout to efficiently locate desired elements as opposed to try touch exploring the region where the the element should be - very tedious. Rationale: There are two ways to implement accessibility focus One is to let accessibility services keep track of it since they have access to the screen content, and another to let the view hierarchy keep track of it. While the first approach would require almost no work on our part it poses several challenges which make it a sub-optimal choice. Having the accessibility focus in the accessibility service would require that service to scrape the window content every time it changes to sync the view tree state and the accessibility focus location. Pretty much the service will have to keep an off screen model of the screen content. This could be quite challenging to get right and would incur performance cost for the multiple IPCs to repeatedly fetch the screen content. Further, keeping virtual accessibility focus (i.e. in the service) would require sync of the input and accessibility focus. This could be challenging to implement right as well. Also, having an unlimited number of accessibility services we cannot guarantee that they will have a proper implementation, if any, to allow users to perform structural navigation of the screen content. Assuming two accessibility services implement structural navigation via accessibility focus, there is not guarantee that they will behave similarly by default, i.e. provide some standard way to navigate the screen content. Also feedback from experienced accessibility researchers, specifically T.V Raman, provides evidence that having virtual accessibility focus creates many issues and it is very hard to get right. Therefore, keeping accessibility focus in the system will avoid keeping an off-screen model in accessibility services, it will always be in sync with the state of the view hierarchy and the input focus. Also this will allow having a default behavior for traversing the screen via this accessibility focus that is consistent in all accessibility services. We provide accessibility services with APIs to override this behavior but all of them will perform screen traversal in a consistent way by default. Behavior: If accessibility is enabled the accessibility focus is the leading one and the input follows it. Putting accessibility focus on a view moves the input focus there. Clearing the accessibility focus of a view, clears the input focus of this view. If accessibility focus is on a view that cannot take input focus, then no other view should have input focus. In accessibility mode we initially give accessibility focus to the topmost view and no view has input focus. This ensures consistent behavior accross all apps. Note that accessibility focus can move hierarchically in the view tree and having it at the root is better than putting it where the input focus would be - at the first input focusable which could be at an arbitrary depth in the view tree. By default not all views are reported for accessibility, only the important ones. A view may be explicitly labeled as important or not for accessibility, or the system determines which one is such - default. Important views for accessibility are all views that are not dumb layout managers used only to arrange their chidren. Since the same content arrangement can be obtained via different combintation of layout managers, such managers cannot be used to reliably determine the application structure. For example, a user should see a list as a list view with several list items and each list item as a text view and a button as opposed to seeing all the layout managers used to arrange the list item's content. By default only important for accessibility views are regared for accessibility purposes. View not regarded for accessibility neither fire accessibility events, nor are reported being on the screen. An accessibility service may request the system to regard all views. If the target SDK of an accessibility services is less than JellyBean, then all views are regarded for accessibility. Note that an accessibility service that requires all view to be ragarded for accessibility may put accessibility focus on any view. Hence, it may implement any navigational paradigm if desired. Especially considering the fact that the system is detecting some standard gestures and delegates their processing to an accessibility service. The default implementation of an accessibility services performs the defualt navigation. bug:5932640 bug:5605641 Change-Id: Ieac461d480579d706a847b9325720cb254736ebe
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
57c7fd5a43237afc5e8ef31a076e862c0c16c328 |
|
24-Feb-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Fixing issues with the AccessibilityNodeInfo cache. 1. Before there were two caches one in the app process that kept track only the ids of infos that were given to a querying client and one in the querying client that holds the infos. This design requires precise sync between the caches. Doing that is somehow complicated since the app has cache for each window and it has to intercept all accessibility events from that window to manage the cache. Each app has to have a cache for each querying client. This approach would guarantee that no infos are fetched twice but due to its stateful nature and the two caches is tricky to implement and adds unnecessary complexity. Now there is only one cache in the client and the apps are stateless. The client is passing flags to the app that are a clue what nodes to prefetch. This approach may occasionally fetch a node twice but it is considerably simpler and stateless from the app perspective - there is only one cache. Fetching a node more than once does not cause much overhead compared to the IPC. Change-Id: Ia02f6fe4f82cff9a9c2e21f4a36747de0f414c6f
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
79311c4af8b54d3cd47ab37a120c648bfc990511 |
|
18-Jan-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Speedup the accessibility window querying APIs and clean up. 1. Now when an interrogating client requires an AccessibilibtyNodeInfo we aggressively prefetch all the predecessors of that node and its descendants. The number of fetched nodes in one call is limited to keep the APIs responsive. The prefetched nodes infos are cached in the client process. The node info cache is invalidated partially or completely based on the fired accessibility events. For example, TYPE_WINDOW_STATE_CHANGED event clears the cache while TYPE_VIEW_FOCUSED removed the focused node from the cache, etc. Note that the cache is only for the currently active window. The ViewRootImple also keeps track of only the ids of the node infos it has sent to each querying process to avoid duplicating work. Usually only one process will query the screen content but we support the general case. Also all the caches are automatically invalidated so not additional bookkeeping is required. This simple strategy leads to 10X improving the speed of the querying APIs. 2. The Monkey and UI test automation framework were registering a raw event listener for accessibility events and hence perform connection and cache management in similar way to an AccessibilityService. This is fragile and requires the implementer to know internal framework stuff. Now the functionality required by the Monkey and the UI automation is encapsulated in a new UiTestAutomationBridge class. To enable this was requited some refactoring of AccessibilityService. 3. Removed the *doSomethiong*InActiveWindow methods from the AccessibilityInteractionClient and the AccessibilityInteractionConnection. The function of these methods is implemented by the not *InActiveWindow version while passing appropriate constants. 4. Updated the internal window Querying tests to use the new UiTestAutomationBridge. 5. If the ViewRootImple was not initialized the querying APIs of the IAccessibilityInteractionConnection implementation were returning immediately without calling the callback with null. This was causing the client side to wait until it times out. Now the client is notified as soon as the call fails. 6. Added a check to guarantee that Views with AccessibilityNodeProvider do not have children. bug:5879530 Change-Id: I3ee43718748fec6e570992c7073c8f6f1fc269b3
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
021078554b902179442a345a9d080a165c3b5139 |
|
04-Oct-2011 |
Svetoslav Ganov <svetoslavganov@google.com> |
Adding APIs to enable reporting virtual view hierarchies to accessibility serivces. Added an interface that is the contract for a client to expose a virtual view hierarchy to accessibility services. Clients impement this interface and set it in the View that is the root of the virtual sub-tree. Adding this finctionality via compostion as opposed to inheritance enables apps to maintain backwards compatibility by setting the accessibility virtual hierarchy provider on the View only if the API version is high enough. bug:5382859 Change-Id: I7e3927b71a5517943c6cb071be2e87fba23132bf
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
8bd69610aafc6995126965d1d23b771fe02a9084 |
|
23-Aug-2011 |
Svetoslav Ganov <svetoslavganov@google.com> |
Intra-process view hierarchy interrogation does not work. The content retrieval APIs are synchronous from a client's perspective but internally they are asynchronous. The client thread calls into the system requesting an action and providing a callback to receive the result after which it waits up to a timeout for that result. The system enforces security and then delegates the request to a given view hierarchy where a message is posted (from a binder thread) describing what to be performed by the main UI thread the result of which it delivered via the mentioned callback. However, the blocked client thread and the main UI thread of the target view hierarchy can be the same one, for example an accessibility service and an activity run in the same process, thus they are executed on the same main thread. In such a case the retrieval will fail since the UI thread that has to process the message describing the work to be done is blocked waiting for a result is has to compute! To avoid this scenario when making a call the client also passes its process and thread ids so the accessed view hierarchy can detect if the client making the request is running in its main UI thread. In such a case the view hierarchy, specifically the binder thread performing the IPC to it, does not post a message to be run on the UI thread but passes it to the singleton interaction client through which all interactions occur and the latter is responsible to execute the message before starting to wait for the asynchronous result delivered via the callback. In this case the expected result is already received so no waiting is performed. bug:5138933 Change-Id: I382e2d8689f5189110226613c2387f553df98bd3
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
eeee4d2c01d3c4ed99e4891dbc75c7de69a803fa |
|
11-Jun-2011 |
Svetoslav Ganov <svetoslavganov@google.com> |
Final polish of the interrogation feature. 1. Added a new event type for notifying client accessibilitiy services for changes in the layout. The event is fired at most once for a given time frame and is delivered to clients only if it originates from the window that can be interrogated. 2. Exposed the findByText functionality in AccessibilityNodeInfo. This is very useful for an accessibility service since it allows searching for something the user knows is on the screen thus avoiding touch exploring the content. Touch exploring is excellent for learning the apps but knowing them search is much faster. 3. Fixed a bug causing an accessibiliby service not to receive the event source in case of more than one service is registered and one of them does not have paermission to interrogate the window. The same event was dispatched to multiple services but if one of them does not have interrogation permission the event is modified to remove the source causing subsequent serivices not to get the later. 4. Moved the getSource setSource methods to AccessibilityRecord instead in AccessibilityEvent. 5. Hiden some protected members in AccessibilityRecod which should not be made public since getters exist. 6. Added the View absolute coordinates in the screen to AccessibilityNodeInfo. This is needed for fast computation of relative positions of views from accessibility - common use case for the later. 7. Fixed a couple of marshalling bugs. 8. Added a test for the object contract of AccessibilityNodeInfo. Change-Id: Id9dc50c33aff441e4c93d25ea316c9bbc4bd7a35
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|
8643aa0179e598e78d938c59035389054535a229 |
|
20-Apr-2011 |
Svetoslav Ganov <svetoslavganov@google.com> |
Interrogation of the view hierarchy from an AccessibilityService. 1. Views are represented as AccessibilityNodeInfos to AccessibilityServices. 2. An accessibility service receives AccessibilityEvents and can ask for its source and gets an AccessibilityNodeInfo which can be used to get its parent and children infos and so on. 3. AccessibilityNodeInfo contains some attributes and actions that can be performed on the source. 4. AccessibilityService can request the system to preform an action on the source of an AccessibilityNodeInfo. 5. ViewAncestor provides an interaction connection to the AccessibiltyManagerService and an accessibility service uses its connection to the latter to interact with screen content. 6. AccessibilityService can interact ONLY with the focused window and all calls are routed through the AccessibilityManagerService which imposes security. 7. Hidden APIs on AccessibilityService can find AccessibilityNodeInfos based on some criteria. These API go through the AccessibilityManagerServcie for security check. 8. Some actions are hidden and are exposes only to eng builds for UI testing. Change-Id: Ie34fa4219f350eb3f4f6f9f45b24f709bd98783c
/frameworks/base/core/java/android/view/accessibility/IAccessibilityInteractionConnection.aidl
|