3ec2e1b94ab502bfa8d5e8750716108f69aa18d2 |
|
09-May-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Remove activation gestures form reported and add a touch explore requesting flag. 1. Delegating activation gestures has several issues that we should decide how to handle if possible before allowing an accessibility service to take over them: A) It is needed that every view than can be clicked or long pressed on reacts to such as a response to calling performClick and performLongPress which is not necessary true since the view may watch the touch events and do its own click long click detection. As a result it may be possible that there are view a user cannot interact with in touch exploration mode but can if not in that mode. B) Clicking or long pressing on a different location in a view may yield different results, for example NumberPicker. Ideally such views have to implement AccessibilityNodeProvide which provider handles correctly the request for click long press on virtual nodes. Some apps however just fire different hover accessibility events when the user is over a specific semantic portion of the view but do not provide virtual nodes. Hence, a user will not be able to interact with such semantic regions but the system can achieve that by sending the click/long click at the precise location in the view that was last touch explored. 2. Adding a flag on accessibility service info to request explore by touch mode. There is no need to put the device in this mode if node of the currently enabled accessibility services supports it. Now the problem is inverted and the service has to explicitly state its capability. 3. Fixing a bug where includeImportantViews was ignored for automation services. Change-Id: I3b29a19f24ab5e26ee29f974bbac2197614c9e2a Conflicts: api/current.txt
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
df549f83810c6fa76be1845363ef69809a30e569 |
|
09-May-2012 |
Guang Zhu <guangzhu@google.com> |
Make UiTestAutomationBridge see non-important views again This problem was introduced in I74df9c24. The intention of the change was still let UiTestAutomationBridge see the non-important views, but there were bugs in the implementation: 1. AccessibilityManagerService was not really updating mIncludeNotImportantViews when mIsAutomation is true 2. Wrong constant is used to set the flag Change-Id: Ia0a2e9ed9720bd0ea3a563e0b492e870a6ec1586
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
ef5889810c32eca9d874503fdc7df6e2a1575993 |
|
08-May-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
DefaultGestureHandlingHelperService should not include non-important views. 1. Since we are using a stateless proxy accessibility service to perform default accessibility gesture handling it shuld not operate against not important views. bug:6422069 Change-Id: I74df9c2415ab3b164d9ac5873f7004c0459e2bfa
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
aa780c110922148a6a4ba06734bb2b0bb8c98f93 |
|
20-Apr-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Adding support for traversing the content of a node info at granularity. 1. A view that creates an accessibility node info may add to the info a list of granularity labels. These are granularities by which the source view can iterate over its content. For example a text view may support character, word link while a web view may additionally support buttons, tables, etc. There are actions on accessibility node info to go to the next/previous at a given granularity which is passesed as an argument. 2. Added Bundle argument to the APIs for performing accessibility actions. This is generic and extensible. bug:5932640 Change-Id: I328cbbb4cddfdee082ab2a8b7ff1bd7477d8d6f9
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
fefd20e927b7252d63acb7bb1852c5188e3c1b2e |
|
20-Apr-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Adding an opt-in mechanism for gesture detection in AccessibilityService. 1. An accessibility service has to explicitly opt in to be notified for gestures by the system. There is only one accessibility service that handles gestures and in case it does not handle a gesture the system performs default handling. This default handling ensures that we have gesture navigation even if no accessibility service would like to participate/customize the interaction model. bug:5932640 Change-Id: Id8194293bd94097b455e9388b68134a45dc3b8fa
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
4213804541a8b05cd0587b138a2fd9a3b7fd9350 |
|
20-Mar-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Accessibility focus - framework Usefulness: Keep track of the current user location in the screen when traversing the it. Enabling structural and directional navigation over all elements on the screen. This enables blind users that know the application layout to efficiently locate desired elements as opposed to try touch exploring the region where the the element should be - very tedious. Rationale: There are two ways to implement accessibility focus One is to let accessibility services keep track of it since they have access to the screen content, and another to let the view hierarchy keep track of it. While the first approach would require almost no work on our part it poses several challenges which make it a sub-optimal choice. Having the accessibility focus in the accessibility service would require that service to scrape the window content every time it changes to sync the view tree state and the accessibility focus location. Pretty much the service will have to keep an off screen model of the screen content. This could be quite challenging to get right and would incur performance cost for the multiple IPCs to repeatedly fetch the screen content. Further, keeping virtual accessibility focus (i.e. in the service) would require sync of the input and accessibility focus. This could be challenging to implement right as well. Also, having an unlimited number of accessibility services we cannot guarantee that they will have a proper implementation, if any, to allow users to perform structural navigation of the screen content. Assuming two accessibility services implement structural navigation via accessibility focus, there is not guarantee that they will behave similarly by default, i.e. provide some standard way to navigate the screen content. Also feedback from experienced accessibility researchers, specifically T.V Raman, provides evidence that having virtual accessibility focus creates many issues and it is very hard to get right. Therefore, keeping accessibility focus in the system will avoid keeping an off-screen model in accessibility services, it will always be in sync with the state of the view hierarchy and the input focus. Also this will allow having a default behavior for traversing the screen via this accessibility focus that is consistent in all accessibility services. We provide accessibility services with APIs to override this behavior but all of them will perform screen traversal in a consistent way by default. Behavior: If accessibility is enabled the accessibility focus is the leading one and the input follows it. Putting accessibility focus on a view moves the input focus there. Clearing the accessibility focus of a view, clears the input focus of this view. If accessibility focus is on a view that cannot take input focus, then no other view should have input focus. In accessibility mode we initially give accessibility focus to the topmost view and no view has input focus. This ensures consistent behavior accross all apps. Note that accessibility focus can move hierarchically in the view tree and having it at the root is better than putting it where the input focus would be - at the first input focusable which could be at an arbitrary depth in the view tree. By default not all views are reported for accessibility, only the important ones. A view may be explicitly labeled as important or not for accessibility, or the system determines which one is such - default. Important views for accessibility are all views that are not dumb layout managers used only to arrange their chidren. Since the same content arrangement can be obtained via different combintation of layout managers, such managers cannot be used to reliably determine the application structure. For example, a user should see a list as a list view with several list items and each list item as a text view and a button as opposed to seeing all the layout managers used to arrange the list item's content. By default only important for accessibility views are regared for accessibility purposes. View not regarded for accessibility neither fire accessibility events, nor are reported being on the screen. An accessibility service may request the system to regard all views. If the target SDK of an accessibility services is less than JellyBean, then all views are regarded for accessibility. Note that an accessibility service that requires all view to be ragarded for accessibility may put accessibility focus on any view. Hence, it may implement any navigational paradigm if desired. Especially considering the fact that the system is detecting some standard gestures and delegates their processing to an accessibility service. The default implementation of an accessibility services performs the defualt navigation. bug:5932640 bug:5605641 Change-Id: Ieac461d480579d706a847b9325720cb254736ebe
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
ae7a51f14349ac070926b406a5b0ab0738231383 |
|
24-Feb-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
UiTestAutomationBridge does not terminate its HandlerThread. 1. Now the thread is terminated in the disconnect() method and also it is made demon since it has no pupose outside the context of the bridge client. bug:6053108 Change-Id: Idc25373fddf501eda4f875fea3e944367e4f04bf
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
57c7fd5a43237afc5e8ef31a076e862c0c16c328 |
|
24-Feb-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Fixing issues with the AccessibilityNodeInfo cache. 1. Before there were two caches one in the app process that kept track only the ids of infos that were given to a querying client and one in the querying client that holds the infos. This design requires precise sync between the caches. Doing that is somehow complicated since the app has cache for each window and it has to intercept all accessibility events from that window to manage the cache. Each app has to have a cache for each querying client. This approach would guarantee that no infos are fetched twice but due to its stateful nature and the two caches is tricky to implement and adds unnecessary complexity. Now there is only one cache in the client and the apps are stateless. The client is passing flags to the app that are a clue what nodes to prefetch. This approach may occasionally fetch a node twice but it is considerably simpler and stateless from the app perspective - there is only one cache. Fetching a node more than once does not cause much overhead compared to the IPC. Change-Id: Ia02f6fe4f82cff9a9c2e21f4a36747de0f414c6f
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
0d04e245534cf777dfaf16dce3c51553837c14ff |
|
21-Feb-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Improving accessibility APIs used for UI automation. 1. UiTestAutomationBridge was accessing the root node in the active window by tracking the accessibility event stream and keeping the last active window changing event. Now the bridge is stateless and the root node is fetched by passing special window and view id with the request to the system. 2. AccessibilityNodeInfos that are cached were not finished, i.e. not sealed, causing exception when trying to access their children or rpedecessors. 3. AccessibilityManagerService was not properly restoring its state after the UI automation bridge disconnects from it. I particular the devices was still in explore by touch mode event if no services are enabled and the sutomation bridge is disconnected. 4. ViewRootImpl for the focused window now fires accessibility events when accessibility is enabled to allow accessibility services to determine the current user location. 5. Several missing null checks in ViewRootImpl are fixed since there were scenraios in which a NPE can occur. 6. Update the internal window content querying tests. 7. ViewRootImpl was firing one extra focus event. bug:6009813 bug:6026952 Change-Id: Ib2e058d64538ecc268f9ef7a8f36ead047868a05
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
b765db590ff824fb5827fb773b14ac4286ae2774 |
|
07-Feb-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Adding a method for retching the root node in UiTestAutomationBridge bug:5974791 Change-Id: I46a8b6a259ba31f27fcd6a71d06dc34efdb4de76
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|
79311c4af8b54d3cd47ab37a120c648bfc990511 |
|
18-Jan-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Speedup the accessibility window querying APIs and clean up. 1. Now when an interrogating client requires an AccessibilibtyNodeInfo we aggressively prefetch all the predecessors of that node and its descendants. The number of fetched nodes in one call is limited to keep the APIs responsive. The prefetched nodes infos are cached in the client process. The node info cache is invalidated partially or completely based on the fired accessibility events. For example, TYPE_WINDOW_STATE_CHANGED event clears the cache while TYPE_VIEW_FOCUSED removed the focused node from the cache, etc. Note that the cache is only for the currently active window. The ViewRootImple also keeps track of only the ids of the node infos it has sent to each querying process to avoid duplicating work. Usually only one process will query the screen content but we support the general case. Also all the caches are automatically invalidated so not additional bookkeeping is required. This simple strategy leads to 10X improving the speed of the querying APIs. 2. The Monkey and UI test automation framework were registering a raw event listener for accessibility events and hence perform connection and cache management in similar way to an AccessibilityService. This is fragile and requires the implementer to know internal framework stuff. Now the functionality required by the Monkey and the UI automation is encapsulated in a new UiTestAutomationBridge class. To enable this was requited some refactoring of AccessibilityService. 3. Removed the *doSomethiong*InActiveWindow methods from the AccessibilityInteractionClient and the AccessibilityInteractionConnection. The function of these methods is implemented by the not *InActiveWindow version while passing appropriate constants. 4. Updated the internal window Querying tests to use the new UiTestAutomationBridge. 5. If the ViewRootImple was not initialized the querying APIs of the IAccessibilityInteractionConnection implementation were returning immediately without calling the callback with null. This was causing the client side to wait until it times out. Now the client is notified as soon as the call fails. 6. Added a check to guarantee that Views with AccessibilityNodeProvider do not have children. bug:5879530 Change-Id: I3ee43718748fec6e570992c7073c8f6f1fc269b3
/frameworks/base/core/java/android/accessibilityservice/UiTestAutomationBridge.java
|