Override default Talkback automatic content grouping and generate a custom contentDescription #33690

fabOnReact · 2022-04-22T12:26:09Z

Summary

The Implementation of the functionality consists of:

Checking that an element has no contentDescription and no text and has other content to announce (role, state, etc.) which causes this issue (for ex. the accessibilityRole is announced before the contentDescription for ex. "Button, My text children component")
If Talkback finds no content to announce on the current node, a custom contentDescription is generated from the child elements that respect the following conditions:

If an AccessibilityNodeInfo is considered "actionable" (which Talkback defines as having clickable=true, longClickable=true, or focusable=true, or having AccessibilityActions for any of those), AND it has some content to read like a contentDescription or text, it will be considered focusable.
If an AccessibilityNodeInfo is considered "actionable" AND it does not have content to read like a contentDescription or text Talkback will parse descendant elements looking for non-focusable descendants to use as content.

implementation of a method getTalkbackDescription to generate the above contentDescription from child elements
over-ride parent contentDescription (accessibilityLabel) with the value returned from getTalkbackDescription

Related notes on Android: Role description is announced before the components text, rather than after #31042. This issue fixes #31042.

Changelog

[Android] [Added] - Override default Talkback automatic content grouping and generate a custom contentDescription

Test Plan

PR Branch
1. Screenreader correctly announcing accessible/non-accessible items (link)
2. Screenreader announces Pressability items (link)
3. Button role is announced after child contentDescription with TouchableNativeFeedback (link)
4. Testing for regressions in Accessibility Actions (link)
5. Screenreader focuses on ScrollView Items (link)
6. Recordings of complete test cases in rn-tester app main and pr branch (link)
9. TouchableOpacity with TextInput child announces contentDescription before the Role (link)
10. One of the child has accessibilityState (hasStateDescription triggers the announcement) (link)
11. One of the child has accessibilityHint (hasText triggers the announcement) (link)

Main
15. The View does not announce the child component Text when accessibilityLabel is missing (automatic content grouping) (link)
8. TouchableOpacity with child EditText annouces placeholder text before and after the role (link)

analysis-bot · 2022-04-22T12:35:43Z

Platform	Engine	Arch	Size (bytes)	Diff
ios	-	universal	n/a	--

Base commit: 9e169da
Branch: main

analysis-bot · 2022-04-22T12:54:13Z

Platform	Engine	Arch	Size (bytes)	Diff
android	hermes	arm64-v8a	7,620,789	+2,735
android	hermes	armeabi-v7a	7,035,043	+2,754
android	hermes	x86	7,920,843	+2,717
android	hermes	x86_64	7,894,463	+2,708
android	jsc	arm64-v8a	9,497,831	+2,429
android	jsc	armeabi-v7a	8,275,452	+2,437
android	jsc	x86	9,435,625	+2,406
android	jsc	x86_64	10,028,609	+2,403

Base commit: 9e169da
Branch: main

blavalla · 2022-04-22T19:01:10Z

I think overall this approach will work, but my concern is that we will be doing this work unnecessarily for 90% of elements that happen to trigger it. As in, Talkback would be doing identical work and an identical outcome, so we might as well delegate to its behavior wherever we can.

It's only in cases where an element has no contentDescription or text and happens to have some other content to announce (role, state, etc.) that this behavior needs to be triggered. I think you can probably add some logic to detect this specific scenario with what you've built already, I just wanted to call out that point #1 isn't as simple as just checking for lack of a contentDescription.

facebook#33690 (comment) The custom contentDescription should be triggered if: - an element has no contentDescription **AND** text - **AND** has other content to announce (role, state, etc.)

the check on the contentDescription is enough also for the accessibilityLabel https://github.com/fabriziobertoglio1987/react-native/blob/e0e162921c183e4203fd64c35b2f1a29c6e1ef4f/ReactAndroid/src/main/java/com/facebook/react/uimanager/BaseViewManager.java#L233

@param

… the announcement) from facebook#33690 (comment) I noticed that getStateDescription returns null even when adding accessibilityState. Calling setStateDescription will update the state description, which is not set via the react accessibilityState or other props. Seems that the method [setViewState](https://github.com/fabriziobertoglio1987/react-native/blob/ad6732aa3576786d37478a729b112031dd94b682/ReactAndroid/src/main/java/com/facebook/react/uimanager/BaseViewManager.java#L254) does not call setStateDescription. I will further investigate tomorrow. <details><summary>View#setStateDescription</summary> ```java /** * Sets the {@link View}'s state description. * * A state description briefly describes the states of the view and is primarily used * for accessibility support to determine how the states of a view should be presented to * the user. It is a supplement to the boolean states (for example, checked/unchecked) and * it is used for customized state description (for example, "wifi, connected, three bars"). * State description changes frequently while content description should change less often. * State description should be localized. For android widgets which have default state * descriptions, app developers can call this method to override the state descriptions. * Setting state description to null restores the default behavior. * * @param stateDescription The state description. * @see #getStateDescription() */ @RemotableViewMethod public void setStateDescription(@nullable CharSequence stateDescription) { if (mStateDescription == null) { if (stateDescription == null) { return; } } else if (mStateDescription.equals(stateDescription)) { return; } mStateDescription = stateDescription; if (!TextUtils.isEmpty(stateDescription) && getImportantForAccessibility() == IMPORTANT_FOR_ACCESSIBILITY_AUTO) { setImportantForAccessibility(IMPORTANT_FOR_ACCESSIBILITY_YES); } if (AccessibilityManager.getInstance(mContext).isEnabled()) { AccessibilityEvent event = AccessibilityEvent.obtain(); event.setEventType(AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED); event.setContentChangeTypes(AccessibilityEvent.CONTENT_CHANGE_TYPE_STATE_DESCRIPTION); sendAccessibilityEventUnchecked(event); } } ``` </details> This is the implementation of TalkBack [hasStateDescription](https://github.com/google/talkback/blob/6c0b475b7f52469e309e51bfcc13de58f18176ff/utils/src/main/java/com/google/android/accessibility/utils/AccessibilityNodeInfoUtils.java#L1672-L1677). The following fields of the view are set with the accessibilityState: - [checkable](https://github.com/fabriziobertoglio1987/react-native/blob/ad6732aa3576786d37478a729b112031dd94b682/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L411) => BOOLEAN_PROPERTY_CHECKABLE - [enabled](https://github.com/fabriziobertoglio1987/react-native/blob/ad6732aa3576786d37478a729b112031dd94b682/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L408) => BOOLEAN_PROPERTY_ENABLED <details><summary>setCheckable</summary> https://github.com/aosp-mirror/platform_frameworks_base/blob/19e53cfdc8a5c6ef45c0adf2dd239576ddce5822/core/java/android/view/accessibility/AccessibilityNodeInfo.java#L2008 ```java /** * Sets whether this node is checkable. * * Note: Cannot be called from an * {@link android.accessibilityservice.AccessibilityService}. * This class is made immutable before being delivered to an AccessibilityService. * * * @param checkable True if the node is checkable. * * @throws IllegalStateException If called from an AccessibilityService. */ public void setCheckable(boolean checkable) { setBooleanProperty(BOOLEAN_PROPERTY_CHECKABLE, checkable); } ``` </details> <details><summary>setEnabled</summary> https://github.com/aosp-mirror/platform_frameworks_base/blob/19e53cfdc8a5c6ef45c0adf2dd239576ddce5822/core/java/android/view/accessibility/AccessibilityNodeInfo.java#L2227 ```java /** * Sets whether this node is enabled. * * Note: Cannot be called from an * {@link android.accessibilityservice.AccessibilityService}. * This class is made immutable before being delivered to an AccessibilityService. * * * @param enabled True if the node is enabled. * * @throws IllegalStateException If called from an AccessibilityService. */ public void setEnabled(boolean enabled) { setBooleanProperty(BOOLEAN_PROPERTY_ENABLED, enabled); } ``` </details> The implementation of `hasStateDescription` is still valid

…ent)

…34245) Summary: Previously published by [grgr-dkrk][2] as [#31296], fixes the following issues: 1) ImportantForAccessibility="no" does not work on Text, Button 2) ImportantForAccessibility="no-hide-descendants" does not work on Text, Button, or ImageBackground. Note: On ImageBackground, focus is prevented on the imageBackground node itself, but focus is not prevented on its descendants. Note: [Button component expected behavior for prop importantForAccessibility][4] >Some components like Button seem like atomic units, but they end up rendering a hierarchy of components for example a wrapper View with a Text inside them. Allowing those descendants to become focusable, breaks the model of these being a single component to consider and forcing no-hide-descendants makes sense here. >The other option is always to render any descendants of these elements with importantForAccessibility="no", so they can never be focusable on their own. This would have the same result, **BUT may potentially cause issues when the descendant content is supposed to automatically get moved up to the focusable ancestor to act as a label** (which is what Talkback does with unfocusable text elements by default). Note: [importantForAccessibility="no" does not allow screenreader focus on nested Text Components with accessibilityRole="link" or inline Images][5] fixes #30850 related #33690 ## Changelog [Android] [Fixed] - adding importantForAccessibility for Text, Button, ImageBackground Pull Request resolved: #34245 Test Plan: 1) testing ImageBackground importantForAccessiblity ([link to test][1]) 2) importantForAccessibility="no" does not allow screenreader focus on nested Text Components with accessibilityRole="link" or inline Images ([link to test][5]) 3) testing ImageBackground importantForAccessiblity ([link to test][6]) 4) Button with importantForAccessibility=no ([link to test][7]) [1]: #31296 (comment) "" [2]: https://github.com/grgr-dkrk "grgr-dkrk" [3]: #31296 "#31296" [4]: #31296 (comment) "expected behaviour with prop importantForAccessibility in Button component" [5]: #30850 (comment) "importantForAccessibility=no does not allow screenreader focus on nested Text Components with accessibilityRole=link or inline Images" [6]: #34245 (comment) "testing ImageBackground importantForAccessiblity" [7]: #34245 (comment) "Button with importantForAccessibility=no" Reviewed By: cipolleschi Differential Revision: D38121992 Pulled By: dmitryrykun fbshipit-source-id: 368b4dcb47d7940274820aa2e39ed5e2ca068821

…acebook#34245) Summary: Previously published by [grgr-dkrk][2] as [facebook#31296], fixes the following issues: 1) ImportantForAccessibility="no" does not work on Text, Button 2) ImportantForAccessibility="no-hide-descendants" does not work on Text, Button, or ImageBackground. Note: On ImageBackground, focus is prevented on the imageBackground node itself, but focus is not prevented on its descendants. Note: [Button component expected behavior for prop importantForAccessibility][4] >Some components like Button seem like atomic units, but they end up rendering a hierarchy of components for example a wrapper View with a Text inside them. Allowing those descendants to become focusable, breaks the model of these being a single component to consider and forcing no-hide-descendants makes sense here. >The other option is always to render any descendants of these elements with importantForAccessibility="no", so they can never be focusable on their own. This would have the same result, **BUT may potentially cause issues when the descendant content is supposed to automatically get moved up to the focusable ancestor to act as a label** (which is what Talkback does with unfocusable text elements by default). Note: [importantForAccessibility="no" does not allow screenreader focus on nested Text Components with accessibilityRole="link" or inline Images][5] fixes facebook#30850 related facebook#33690 ## Changelog [Android] [Fixed] - adding importantForAccessibility for Text, Button, ImageBackground Pull Request resolved: facebook#34245 Test Plan: 1) testing ImageBackground importantForAccessiblity ([link to test][1]) 2) importantForAccessibility="no" does not allow screenreader focus on nested Text Components with accessibilityRole="link" or inline Images ([link to test][5]) 3) testing ImageBackground importantForAccessiblity ([link to test][6]) 4) Button with importantForAccessibility=no ([link to test][7]) [1]: facebook#31296 (comment) "" [2]: https://github.com/grgr-dkrk "grgr-dkrk" [3]: facebook#31296 "facebook#31296" [4]: facebook#31296 (comment) "expected behaviour with prop importantForAccessibility in Button component" [5]: facebook#30850 (comment) "importantForAccessibility=no does not allow screenreader focus on nested Text Components with accessibilityRole=link or inline Images" [6]: facebook#34245 (comment) "testing ImageBackground importantForAccessiblity" [7]: facebook#34245 (comment) "Button with importantForAccessibility=no" Reviewed By: cipolleschi Differential Revision: D38121992 Pulled By: dmitryrykun fbshipit-source-id: 368b4dcb47d7940274820aa2e39ed5e2ca068821

blavalla · 2022-08-06T00:15:10Z

packages/rn-tester/js/examples/Accessibility/AccessibilityExample.js

+          </TouchableNativeFeedback>
+        </RNTesterBlock>
+
+        <RNTesterBlock title="The child is not TextInput, the contentDescription is not empty and does not have nodeText">


Remove duplicate example (this is the same as the previous one)

commit ddfb345

blavalla · 2022-08-06T00:17:02Z

packages/rn-tester/js/examples/Accessibility/AccessibilityExample.js

+          </TouchableNativeFeedback>
+        </RNTesterBlock>
+
+        <RNTesterBlock title="One of the child has accessibilityState (hasStateDescription triggers the announcement)">


This example feels a bit odd to me. The state itself is not concatenated into the announcement, which makes sense, as it would lead to potentially announcing conflicting state if two child elements had separate states like "checked" and "unchecked", but because of this I'm not sure what value this example is providing. If there was no state here and just the label it would work the exact same way.

I think we can probably just remove this one and maybe replace it with one where the parent element has a state and making sure that state is reflected even when the children are combined into it.

Thanks, @blavalla. I did the changes with commit ddfb345

I improved and moved the example before One of the children has accessibilityLabel, role, state, and accValue

Talkback only pulls the child's contentDescription or text but does not include the child's accessibilityState or accessibilityRole. TalkBack avoids announcements of conflicting states or roles (for example, 'button' and 'slider').

blavalla · 2022-08-06T00:37:11Z

packages/rn-tester/js/examples/Accessibility/AccessibilityExample.js

+        <RNTesterBlock title="Parent and children have different role">
+          <TouchableNativeFeedback
+            accessible={true}
+            importantForAccessibility="yes"


I don't think importantForAccessibility="yes" is necessary for any of these examples, so lets remove it on all of them just to make it clear that this isn't a prequalification for this feature working.

commit ddfb345

This example feels a bit odd to me. The state itself is not concatenated into the announcement, which makes sense, as it would lead to potentially announcing conflicting state if two child elements had separate states like "checked" and "unchecked", but because of this I'm not sure what value this example is providing. If there was no state here and just the label it would work the exact same way. I think we can probably just remove this one and maybe replace it with one where the parent element has a state and making sure that state is reflected even when the children are combined into it. facebook#33690 (comment) facebook#33690 (comment) I don't think importantForAccessibility="yes" is necessary for any of these examples, so lets remove it on all of them just to make it clear that this isn't a prequalification for this feature working. facebook#33690 (comment) Remove duplicate example (this is the same as the previous one) facebook#33690 (comment)

facebook-github-bot · 2022-08-31T17:38:18Z

@blavalla has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

github-actions · 2022-12-02T05:35:41Z

This pull request was successfully merged by @fabriziobertoglio1987 in 759056b.

^{When will my fix make it into a release? | Upcoming Releases}

…ustom contentDescription (facebook#33690) Summary: The Implementation of the functionality consists of: 1) Checking that an element has no contentDescription and no text and has other content to announce (role, state, etc.) which causes this issue (for ex. the accessibilityRole is announced before the contentDescription for ex. "Button, My text children component") 2) If Talkback finds no content to announce on the current node, a custom contentDescription is generated from the child elements that respect the following conditions: >If an AccessibilityNodeInfo is considered "actionable" (which Talkback defines as having clickable=true, longClickable=true, or focusable=true, or having AccessibilityActions for any of those), AND it has some content to read like a contentDescription or text, it will be considered focusable. >If an AccessibilityNodeInfo is considered "actionable" AND it does not have content to read like a contentDescription or text Talkback will parse descendant elements looking for non-focusable descendants to use as content. 3) implementation of a method getTalkbackDescription to generate the above contentDescription from child elements 4) over-ride parent contentDescription (accessibilityLabel) with the value returned from getTalkbackDescription Related [notes on Android: Role description is announced before the components text, rather than after facebook#31042]. This issue fixes [facebook#31042]. ## Changelog [Android] [Added] - Override default Talkback automatic content grouping and generate a custom contentDescription Pull Request resolved: facebook#33690 Test Plan: **PR Branch** [1]. Screenreader correctly announcing accessible/non-accessible items ([link][1]) [2]. Screenreader announces Pressability items ([link][2]) [3]. Button role is announced after child contentDescription with TouchableNativeFeedback ([link][3]) [4]. Testing for regressions in Accessibility Actions ([link][4]) [5]. Screenreader focuses on ScrollView Items ([link][5]) [6]. Recordings of complete test cases in rn-tester app main and pr branch ([link][6]) [9]. TouchableOpacity with TextInput child announces contentDescription before the Role ([link][9]) [10]. One of the child has accessibilityState (hasStateDescription triggers the announcement) ([link][10]) [11]. One of the child has accessibilityHint (hasText triggers the announcement) ([link][11]) **Main** [15]. The View does not announce the child component Text when accessibilityLabel is missing (automatic content grouping) ([link][15]) [8]. TouchableOpacity with child EditText annouces placeholder text before and after the role ([link][8]) [1]: fabOnReact/react-native-notes#14 (comment) "Screenreader correctly announcing accessible/non-accessible items" [2]: fabOnReact/react-native-notes#14 (comment) "Screenreader announces Pressability items" [3]: fabOnReact/react-native-notes#14 (comment) "Button role is announced after child contentDescription" [4]: fabOnReact/react-native-notes#14 (comment) "Testing for regressions in Accessibility Actions" [5]: fabOnReact/react-native-notes#14 (comment) "Screenreader focuses on ScrollView Items" [6]: fabOnReact/react-native-notes#14 (comment) "Recordings of complete test cases in rn-tester app main and pr branch" [7]: fabOnReact/react-native-notes#14 (comment) "TouchableOpacity with child EditText annouces Button role before the child contentDescription" [8]: fabOnReact/react-native-notes#14 (comment) "TouchableOpacity with child EditText annouces placholder text before and after the role" [9]: fabOnReact/react-native-notes#14 (comment) "TouchableOpacity with TextInput child announces contentDescription before the Role" [10]: fabOnReact/react-native-notes#14 (comment) "One of the child has accessibilityState (hasStateDescription triggers the announcement)" [11]: fabOnReact/react-native-notes#14 (comment) "One of the child has accessibilityHint (hasText triggers the announcement)" [15]: fabOnReact/react-native-notes#14 (comment) "The View does not announce the child component Text when accessibilityLabel is missing (automatic content grouping)" [50]: facebook#31042 "Android: Role description is announced before the components text, rather than after facebook#31042" [51]: fabOnReact/react-native-notes#14 "notes on Android: Role description is announced before the components text, rather than after facebook#31042" Reviewed By: cipolleschi Differential Revision: D39177512 Pulled By: blavalla fbshipit-source-id: 6bd0fba9c347bc14b3839e903184c86d2bcab3d2

fabOnReact added 12 commits April 7, 2022 04:06

draft

05a2bc3

draft

4ffd15d

Merge branch 'main' into automatic-content-grouping

5e1622f

draft

28f86a3

Merge branch 'main' into automatic-content-grouping

963cd17

draft solution to announce content

69473de

remove previous draft solution

bdabc4c

draft commit

ae66001

remove getRole logic

7c287a4

adding comments to methods

6d73cd4

reorganize methods alph order

2a3a836

revert changes to ButtonExample

982fb20

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 22, 2022

react-native-bot added Platform: Android Android applications. Type: Enhancement A new feature or enhancement of an existing feature. labels Apr 22, 2022

fabOnReact mentioned this pull request Apr 25, 2022

Android: Role description is announced before the components text, rather than after #31042 fabOnReact/react-native-notes#14

Closed

fabOnReact added 11 commits April 25, 2022 18:51

Conditions that should trigger the custom contentDescription

e0e1629

facebook#33690 (comment) The custom contentDescription should be triggered if: - an element has no contentDescription **AND** text - **AND** has other content to announce (role, state, etc.)

remove duplicate logic and add condition

07bac60

avoid setContentDescription if talkback returns null

821e623

adding GRID role to RADelegate

35190c0

adding TouchableNativeFeedback with non-acc child example

986b10b

adding TouchableNativeFeedback with non-acc child second example

28ed277

adding @nullable to description variable

e1521cf

adding accessibilityAction to Text example

90dd4a8

Merge branch 'main' into automatic-content-grouping

5e7aaf3

Merge branch 'main' into automatic-content-grouping

1f06997

fabOnReact added 5 commits June 7, 2022 09:21

example - getStateDescription returns allways null

d5f4c9c

One of the child has accessibilityHint (hasText trigers the announcem…

087e642

…ent)

remove import of Switch in AccessibilityExample

51f23f0

fix AccessibilityExample actions

8b84521

fabOnReact marked this pull request as ready for review June 8, 2022 14:15

fabOnReact mentioned this pull request Jul 4, 2022

Android: Role description is announced before the components text, rather than after #31042

Closed

This was referenced Jul 25, 2022

adding importantForAccessibility for Text, Button, ImageBackground #34245

Closed

Android: "accessible" prop issues. #30851

Closed

fabOnReact mentioned this pull request Aug 5, 2022

Android: using AccessibilityNodeInfo#addAction to announce Expandable/Collapsible State #34353

Closed

Merge branch 'main' into automatic-content-grouping

dc0cc5a

fabOnReact mentioned this pull request Aug 12, 2022

1/2 TextInput accessibilityErrorMessage (Talkback, Android) #33468

Closed

blavalla suggested changes Aug 29, 2022

View reviewed changes

fabOnReact self-assigned this Aug 30, 2022

Merge branch 'main' into automatic-content-grouping

eac460e

fabOnReact marked this pull request as draft August 30, 2022 09:55

fabOnReact marked this pull request as ready for review August 30, 2022 11:41

improving examples titles

11c4b00

blavalla approved these changes Aug 31, 2022

View reviewed changes

facebook-github-bot closed this in 759056b Dec 2, 2022

github-actions bot added the Merged This PR has been merged. label Dec 2, 2022

fabOnReact added the Accessibility label Jan 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Override default Talkback automatic content grouping and generate a custom contentDescription #33690

Override default Talkback automatic content grouping and generate a custom contentDescription #33690

fabOnReact commented Apr 22, 2022 •

edited

Loading

analysis-bot commented Apr 22, 2022 •

edited

Loading

analysis-bot commented Apr 22, 2022 •

edited

Loading

blavalla commented Apr 22, 2022

blavalla Aug 6, 2022

fabOnReact Aug 30, 2022

blavalla Aug 6, 2022

fabOnReact Aug 30, 2022 •

edited

Loading

blavalla Aug 6, 2022

fabOnReact Aug 30, 2022

facebook-github-bot commented Aug 31, 2022

github-actions bot commented Dec 2, 2022

Override default Talkback automatic content grouping and generate a custom contentDescription #33690

Override default Talkback automatic content grouping and generate a custom contentDescription #33690

Conversation

fabOnReact commented Apr 22, 2022 • edited Loading

Summary

Changelog

Test Plan

analysis-bot commented Apr 22, 2022 • edited Loading

analysis-bot commented Apr 22, 2022 • edited Loading

blavalla commented Apr 22, 2022

blavalla Aug 6, 2022

Choose a reason for hiding this comment

fabOnReact Aug 30, 2022

Choose a reason for hiding this comment

blavalla Aug 6, 2022

Choose a reason for hiding this comment

fabOnReact Aug 30, 2022 • edited Loading

Choose a reason for hiding this comment

blavalla Aug 6, 2022

Choose a reason for hiding this comment

fabOnReact Aug 30, 2022

Choose a reason for hiding this comment

facebook-github-bot commented Aug 31, 2022

github-actions bot commented Dec 2, 2022

fabOnReact commented Apr 22, 2022 •

edited

Loading

analysis-bot commented Apr 22, 2022 •

edited

Loading

analysis-bot commented Apr 22, 2022 •

edited

Loading

fabOnReact Aug 30, 2022 •

edited

Loading