9b278112467581e2b8d99f29cf48b9fbef9cc53f |
|
04-Jan-2016 |
Anna Galusza <agalusza@google.com> |
Add API for IME control by Accessibility Services. Change-Id: I3bb806cf420e0551a2c9ef97d95613f73e362df9
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
a6b64f5099b7be6e8384958d8bcddb97bb06ec93 |
|
05-Dec-2015 |
Phil Weaver <pweaver@google.com> |
Gesture dispatch from accessibility services. Add public APIs to describe gestures and dispatch them from an accessibility service. Added a new capability that services must declare to have this capability. Bug: 22514086 Change-Id: I9bff2d9335f0310115112d14b7ed033a6d6c2393
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
214fb68767502f5fede643a062c1dc5975d75b27 |
|
17-Nov-2015 |
Alan Viverette <alanv@google.com> |
APIs for querying and controlling display magnification Also separates magnification state and touch event handling. Moves callbacks for window manager changes and display state changes into the magnification controller. Bug: 22718911 Change-Id: I3a8ba060a07d8f1f51856855a5f85601766fd45d
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
3a5c721072c60c7ed9c8a95d0a65d0e3cb4eb9bb |
|
14-Oct-2014 |
Svetoslav <svetoslavganov@google.com> |
APIs for an accessibility service to put interaction tracking overlays. An accessibility service may register to observe the interactive windows on the primary display. These windows are the one that has input focus and ones a sighted user can touch. It is sometimes beneficial for an accessibility service to overlay a window to intercept user interaction and based on that introspect and perform an action on the windows that are on the screen. This is problematic as overlaying a full screen window that is touchable prevents the accessibility service to introspect the content under this window. This change adds a special type of window that only an accessibility service can place which does not affect what an accessibility service can "see" on the screen. Hence, even putting such a window full screen the service will be able to interact with the other interactive windows it covers. Change-Id: I053ccc3a5c6360a98dc40bdb172b54dab35d8b31
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
a4725efd0bfa52cbddf6ca587d37fc4ebcbfaf72 |
|
24-Jul-2014 |
Svetoslav <svetoslavganov@google.com> |
Improve the window query API performamce. We are caching the window data in the accessibility service process. When windows change we were sending the dalta of the windows the service knows about. To make this work when the app asked for all windows we had to call into the system as new windows may have appeared. This was slow. Now we are telling the service some windows change and if it gets the windows we cache them. We call into the system only on a cache miss and evict all windows from the cache on window change event. We do not evict the nodes of the window as the former may have just moved. If views in a window change they fire accessibility events that trigger the correct eviction. Change-Id: I586a72a2497b0d44a75288fa758e7e88817f3300
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
8e3feb15c5aec2c72b0ef120a1da325e1e8f0dda |
|
24-Feb-2014 |
Svetoslav <svetoslavganov@google.com> |
Added accessibility APIs for introspecting interactive windows. 1. The old introspection model was allowing querying only the active window which is the one the user is touching or the focused one if no window is touched. This was limiting as auto completion drop downs were not inspectable, there was not way to know when the IME toggles, non-focusable windows were not inspectable if the user taps them as until a screen-reader starts introspecting the users finger is up, accessibility focus was limited to only one window and the user couldn't use gestures to visit the whole UI, and other things I can't remember right now. The new APIs allow getting all interactive windows, i.e. ones that a sighted user can interact with. This prevents an accessibility service from interacting with content a sighter user cannot. The list of windows can be obtained from an accessibility service or the host window from an accessibility node info. Introspecting windows obey the same rules for introspecting node, i.e. the service has to declare this capability in its manifest. When some windows change accessibility services receive a new type of event. Initially the types of windows is very limited. We provide the bounds in screen, layer, and some other properties which are enough for a client to determined the spacial and hierarchical relationship of the windows. 2. Update the documentation in AccessibilityService for newer event types. 3. LongArray was not removing elements properly. 4. Composite accessibility node ids were not properly constructed as they are composed of two ints, each taking 32 bits. However, the values for undefined were -1 so composing a 64 long from -1, -1 prevents from getting back these values when unpacking. 5. Some apps were generating inconsistent AccessibilityNodeInfo tree. Added a check that enforces such trees to be well formed on dev builds. 6. Removed an necessary code for piping the touch exploration state to the policy as it should just use the AccessibilityManager from context. 7. When view's visibility changed it was not firing an event to notify clients it disappeared/appeared. Also ViewGroup was sending accessibility events for changes if the view is included for accessibility but this is wrong as there may be a service that want all nodes, hence events from them. The accessibility manager service takes care of delivering events from not important for accessibility nodes only to services that want such. 8. Several places were asking for prefetching of sibling but not predecessor nodes which resulted in prefetching of unconnected subtrees. 9. The local AccessibilityManager implementation was relying on the backing service being ready when it is created but it can be fetched from a context before that. If that happens the local manager was in a broken state forever. Now it is more robust and starts working properly once the backing service is up. Several places were lacking locking. bug:13331285 Change-Id: Ie51166d4875d5f3def8d29d77973da4b9251f5c8
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
c4fccd183f1bb47a027bb303af5e65bec2f68b1b |
|
09-Apr-2013 |
Svetoslav <svetoslavganov@google.com> |
Adding APIs for an accessibility service to intercept key events. Now that we have gestures which are detected by the system and interpreted by an accessibility service, there is an inconsistent behavior between using the gestures and the keyboard. Some devices have both. Therefore, an accessibility service should be able to interpret keys in addition to gestures to provide consistent user experience. Now an accessibility service can expose shortcuts for each gestural action. This change adds APIs for an accessibility service to observe and intercept at will key events before they are dispatched to the rest of the system. The service can return true or false from its onKeyEvent to either consume the event or to let it be delivered to the rest of the system. However, the service will *not* be able to inject key events or modify the observed ones. Previous ideas of allowing the service to say it "tracks" the event so the latter is not delivered to the system until a subsequent event is either "handled" or "not handled" will not work. If the service tracks a key but no other key is pressed essentially this key is not delivered to the app and at potentially much later point this stashed event will be delivered in maybe a completely different context.The correct way of implementing shortcuts is a combination of modifier keys plus some other key/key sequence. Key events already contain information about which modifier keys are down as well as the service can track them as well. bug:8088812 Change-Id: I81ba9a7de9f19ca6662661f27fdc852323e38c00
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
152e9bb81aa5b2ab4637f4b2dae04b3ce89fa891 |
|
13-Oct-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Refactoring of the screen magnification feature. 1. The screen magnification feature was implemented entirely as a part of the accessibility manager. To achieve that the window manager had to implement a bunch of hooks for an external client to observe its internal state. This was problematic since it dilutes the window manager interface and allows code that is deeply coupled with the window manager to reside outside of it. Also the observer callbacks were IPCs which cannot be called with the window manager's lock held. To avoid that the window manager had to post messages requesting notification of interested parties which makes the code consuming the callbacks to run asynchronously of the window manager. This causes timing issues and adds unnecessary complexity. Now the magnification logic is split in two halves. The first half that is responsible to track the magnified portion of the screen and serve as a policy which windows can be magnified and it is a part of the window manager. This part exposes higher level APIs allowing interested parties with the right permissions to control the magnification of a given display. The APIs also allow a client to be registered for callbacks on interesting changes such as resize of the magnified region, etc. This part servers as a mediator between magnification controllers and the window manager. The second half is a controller that is responsible to drive the magnification state based on touch interactions. It also presents a highlight when magnified to suggest the magnified potion of the screen. The controller is responsible for auto zooming out in case the user context changes - rotation, new actitivity. The controller also auto pans if a dialog appears and it does not interesect the magnified frame. bug:7410464 2. By design screen magnification and touch exploration work separately and together. If magnification is enabled the user sees a larger version of the widgets and a sub section of the screen content. Accessibility services use the introspection APIs to "see" what is on the screen so they can speak it, navigate to the next item in response to a gesture, etc. Hence, the information returned to accessibility services has to reflect what a sighted user would see on the screen. Therefore, if the screen is magnified we need to adjust the bounds and position of the infos describing views in a magnified window such that the info bounds are equivalent to what the user sees. To improve performance we keep accessibility node info caches in the client process. However, when magnification state changes we have to clear these caches since the bounds of the cached infos no longer reflect the screen content which just got smaller or larger. This patch propagates not only the window scale as before but also the X/Y pan and the bounds of the magnified portion of the screen to the introspected app. This information is used to adjust the bounds of the node infos coming from this window such that the reported bounds are the same as the user sees not as the app thinks they are. Note that if magnification is enabled we zoom the content and pan it along the X and Y axis. Also recomputed is the isVisibleToUser property of the reported info since in a magnified state the user sees a subset of the window content and the views not in the magnified viewport should be reported as not visible to the user. bug:7344059 Change-Id: I6f7832c7a6a65c5368b390eb1f1518d0c7afd7d2
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
7b1e0c7046abefc0b40884b36197c8a803d9cf6d |
|
13-May-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Removing default accessibility gesture handling. 1. The initial design was to have some accessibility gestures being handled by the system if the gesture handling access service does not consume the gesture. However, we are not sure what a good default is and once we add a default handler we cannot remove it since people may rely on it. Thus, we take the simples approach and let the accessibility service handle the gestures. If no gestures are handled the system will work in explore by touch as before. bug:5932640 Change-Id: I865a83549fa03b0141d27ce9713e9b7bb45a57b4
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
fefd20e927b7252d63acb7bb1852c5188e3c1b2e |
|
20-Apr-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Adding an opt-in mechanism for gesture detection in AccessibilityService. 1. An accessibility service has to explicitly opt in to be notified for gestures by the system. There is only one accessibility service that handles gestures and in case it does not handle a gesture the system performs default handling. This default handling ensures that we have gesture navigation even if no accessibility service would like to participate/customize the interaction model. bug:5932640 Change-Id: Id8194293bd94097b455e9388b68134a45dc3b8fa
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|
4213804541a8b05cd0587b138a2fd9a3b7fd9350 |
|
20-Mar-2012 |
Svetoslav Ganov <svetoslavganov@google.com> |
Accessibility focus - framework Usefulness: Keep track of the current user location in the screen when traversing the it. Enabling structural and directional navigation over all elements on the screen. This enables blind users that know the application layout to efficiently locate desired elements as opposed to try touch exploring the region where the the element should be - very tedious. Rationale: There are two ways to implement accessibility focus One is to let accessibility services keep track of it since they have access to the screen content, and another to let the view hierarchy keep track of it. While the first approach would require almost no work on our part it poses several challenges which make it a sub-optimal choice. Having the accessibility focus in the accessibility service would require that service to scrape the window content every time it changes to sync the view tree state and the accessibility focus location. Pretty much the service will have to keep an off screen model of the screen content. This could be quite challenging to get right and would incur performance cost for the multiple IPCs to repeatedly fetch the screen content. Further, keeping virtual accessibility focus (i.e. in the service) would require sync of the input and accessibility focus. This could be challenging to implement right as well. Also, having an unlimited number of accessibility services we cannot guarantee that they will have a proper implementation, if any, to allow users to perform structural navigation of the screen content. Assuming two accessibility services implement structural navigation via accessibility focus, there is not guarantee that they will behave similarly by default, i.e. provide some standard way to navigate the screen content. Also feedback from experienced accessibility researchers, specifically T.V Raman, provides evidence that having virtual accessibility focus creates many issues and it is very hard to get right. Therefore, keeping accessibility focus in the system will avoid keeping an off-screen model in accessibility services, it will always be in sync with the state of the view hierarchy and the input focus. Also this will allow having a default behavior for traversing the screen via this accessibility focus that is consistent in all accessibility services. We provide accessibility services with APIs to override this behavior but all of them will perform screen traversal in a consistent way by default. Behavior: If accessibility is enabled the accessibility focus is the leading one and the input follows it. Putting accessibility focus on a view moves the input focus there. Clearing the accessibility focus of a view, clears the input focus of this view. If accessibility focus is on a view that cannot take input focus, then no other view should have input focus. In accessibility mode we initially give accessibility focus to the topmost view and no view has input focus. This ensures consistent behavior accross all apps. Note that accessibility focus can move hierarchically in the view tree and having it at the root is better than putting it where the input focus would be - at the first input focusable which could be at an arbitrary depth in the view tree. By default not all views are reported for accessibility, only the important ones. A view may be explicitly labeled as important or not for accessibility, or the system determines which one is such - default. Important views for accessibility are all views that are not dumb layout managers used only to arrange their chidren. Since the same content arrangement can be obtained via different combintation of layout managers, such managers cannot be used to reliably determine the application structure. For example, a user should see a list as a list view with several list items and each list item as a text view and a button as opposed to seeing all the layout managers used to arrange the list item's content. By default only important for accessibility views are regared for accessibility purposes. View not regarded for accessibility neither fire accessibility events, nor are reported being on the screen. An accessibility service may request the system to regard all views. If the target SDK of an accessibility services is less than JellyBean, then all views are regarded for accessibility. Note that an accessibility service that requires all view to be ragarded for accessibility may put accessibility focus on any view. Hence, it may implement any navigational paradigm if desired. Especially considering the fact that the system is detecting some standard gestures and delegates their processing to an accessibility service. The default implementation of an accessibility services performs the defualt navigation. bug:5932640 bug:5605641 Change-Id: Ieac461d480579d706a847b9325720cb254736ebe
/frameworks/base/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
|