Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Android] Implement setting content sensitivity on Android #158473

Open
wants to merge 113 commits into
base: master
Choose a base branch
from

Conversation

camsim99
Copy link
Contributor

@camsim99 camsim99 commented Nov 11, 2024

Overview

Implements setting content sensitivity on Android via a new widget called SensitiveContent that currently will only function on Android.

How it's implemented

There are three different content sensitivity levels: autoSensitive, sensitive, notSensitive. All except autoSensitive behave as their Android counterpart level behaves (see links on the levels); for Flutter, autoSensitive will behave the same as notSensitive though it is implemented as if it works (see #160879 for more details.

In any given Flutter view, each SensitiveContent widget sets a content sensitivity level and there may be multiple SensitiveContent widgets in the widget tree at a time, but only the most severe content sensitivity level in the tree will be maintained in order to ensure secure content remains obscured during screen projection. This means that even widgets that are not visible may still impact the overall content sensitivity of the entire Flutter view (see #160051 for more details).

Under the hood, the implementation uses method channels. Theoretically, this could cause a delay in setting the content sensitivity level and a Flutter view being obscured during screen projection on time, so we will investigate using JNIgen in the future (see #160050 for more details). Over the method channel, the framework will send a Flutter view ID and content sensitivity level to the engine to set the expected content sensitivity level for the view.

Timeline

Required for feature shipping

  1. Land this PR with the method channel implementation of content sensitivity.
  2. Convert this PR to use JNIgen: [Android] Modify sensitive content support implementation to use JNIgen #160050. Verify no frames are dropped, revealing sensitive content during media projection: [Android] Verify SensitiveContent widget potential frame drops are not visible #164820.
  3. Verify dropped frame issue is not present: [Android] Verify SensitiveContent widget potential frame drops are not visible #164820
  4. Ensure every FlutterView has a unique ID for the native SensitiveContentPlugin to identify them by: [Android] Make FlutterView IDs instance variables instead of a static one #162685

Stretch goals

  1. Make the SensitiveContent widget sensitive to the visibility of child widgets: [Android] Un-mark sensitive content when it is not visible on screen #160051
  2. Implement autoSensitivity mode: [Android] Investigate implementing autoSensitive sensitive content mode #160879
  3. Investigate backwards compatibility for APIs < 35: [Android] Consider implementing backwards compatibility for sensitive content protection #159208

Open questions

  • Is it okay to hard-code the FlutterActivity and FlutterFragment view IDs? Does this impact add-to-app in any way? I assume we would need extenders of those classes to use a different view ID than the hardcoded one. Approach not used.
  • Should we expose autoSensitive for now? If so, what behavior should it have? Exposing autoSensitive as if it works.

One way to test this PR (rough steps)

  1. Create a Flutter app that has some number of SensitiveContent widgets.
  2. Run the app on an emulator/device that runs API 35 and has the Google meets app downloaded.
  3. Start a Google meets meeting and screen share (works with full screen and single app mode).
  4. Join the Google meets meeting from another device and witness a blacked out screen if content has been marked sensitive.

Pre-launch Checklist

@github-actions github-actions bot added the framework flutter/packages/flutter repository. See also f: labels. label Nov 11, 2024
@camsim99 camsim99 changed the title [Proof of concept] Implement setting content sensitivity on Android Implement setting content sensitivity on Android Nov 20, 2024
@camsim99 camsim99 changed the title Implement setting content sensitivity on Android [Android] Implement setting content sensitivity on Android Nov 20, 2024
@github-actions github-actions bot added the engine flutter/engine repository. See also e: labels. label Dec 26, 2024
Copy link
Contributor

@justinmc justinmc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments and questions about error handling and some edge cases, but otherwise this is looking good. I like the overall approach.


@override
Widget build(BuildContext context) {
return FutureBuilder<void>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see, no that's a great idea actually. Looks good as-is.


// Re-register SensitiveContent widget if the sensitivity level changes.
_sensitiveContentRegistrationFuture = SensitiveContentHost.register(widget.sensitivityLevel);
SensitiveContentHost.unregister(oldWidget.sensitivityLevel);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the sensitivity changes, I think we should be sure that:

  1. Inside SensitiveContentHost, the platform never gets called twice back-to-back (register/unregister).
  2. The build method hides the child until reregistration is finished.

Can you double check that both of those are working properly and consider testing them?

For 1 You might need a SensitiveContentHost.reregister or .update or something.

For 2 you might need to call setState here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a test that I think covers these! I'll think on other cases to test as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm that 1 is pretty thoroughly tested in sensitive_content_host_test.dart and I added new tests for 2.

@override
void initState() {
super.initState();
_sensitiveContentRegistrationFuture = SensitiveContentHost.register(widget.sensitivityLevel);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a follow-up from the discussion about this call failing (https://github.com/flutter/flutter/pull/158473/files#r1976045506), is there anything we should do for error handling here? Currently, I think if this fails then the app will crash. Would it be better to just log an error and display child without doing anything for SensitiveContent?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the calls to setContentSensitivity to log an error!

}

// Set content sensitivity level as desiredSensitivityLevel and update stored data.
_sensitiveContentService.setContentSensitivity(desiredSensitivityLevel);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently I think if this call fails then the whole app will crash right? I left a comment about possibly recovering from this in the widget.

Assuming we don't let the app crash, then after the call fails, _contentSensitivitySetting will have been updated but calculatedContentSensitivityLevel will not. You're saying that's correct, because _contentSensitivitySetting represents the widget tree, and calculatedContentSensitivityLevel represents what we think Android has it set to?

@camsim99
Copy link
Contributor Author

@chunhtai I need to do some minor cleanup + add comments, but if you want to take a look tomorrow and give another review PTAL :)

setContentSensitivityCompleter.complete();

// Delay added to ensure that the SensitiveContent widget re-registration completes.
await Future<void>.delayed(const Duration(milliseconds: 500), () async {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know where is the async operation this is waiting on? It will be better if we wait for a completer future instead of a delay

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I thought this was needed to make sure that re-registering the SensitiveContent widget happens, but it seems to pass without it. Let's see what happens in CI because I did see a failure before.

We use a completer to complete the fake async method call, but I thought we needed to wait for the Future that the FutureBuilder relies on to finish to check that the actual SensitiveContent widget child is built. I thought we couldn't use a Completer for this without faking/mocking the Future that the FutureBuilder relies on.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a pump should be enough since unittest uses a fake async. Unless you are introducing scheduleTask or other type a random delay future in the widget, we shouldn't need to wait a real future in unittest. and even then, we should expose a future for test to await in the await tester.runAsync

@camsim99 camsim99 requested a review from chunhtai March 20, 2025 19:08
Copy link
Contributor

@chunhtai chunhtai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is widget is marked as not production ready and is unreachable in common way. None of my comments are blocking, but I hope we can address them before they become public

default:
// Signal to Flutter framework that the embedder does not recognize
// the content sensitivity mode queried.
return UNKNOWN_CONTENT_SENSITIVITY;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this return error result instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. The idea here was that it a user calls getContentSensitivity and an unknown content sensitivity is returned by Android, we should not crash because the user may have no idea that an unknown one was returned, e.g. if we bump to Android 16 and the default content sensitivity changes. Instead, we return unknown to signal to the framework side to handle this accordingly.

I'm sure we could throw an error to accomplish the same thing but I think using the unknown value is a bit clearer in terms of our intent.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm.

Future<ContentSensitivity> getContentSensitivity() async {
String? result;
try {
result = await sensitiveContentChannel.invokeMethod<String>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we using string intead of index of enum to communicate the sensitivity level?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right now the communication is the following.

andoird (int) -> converting to String -> channel -> converting string to enum -> dart(enum)

dart(enum) -> converting to String -> channel -> converting string to int -> andoird (int)

we could just do the following

android (int) -> converting to enum int in dart -> channel -> recast int to enum -> dart(enum)

dart(enum) -> recast to int -> channel -> converting to android int -> andoird (int)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this the same level of work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I think the Strings are more readable for the serialization/deserialization done on the platform side.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If using string, you will do 2 string to int conversion and sending string through channel is slightly more costly.

If using int, you will do 1 Android int to dart enum conversion.

You can get the same level of readability when using enum. For example the semanticsFlag, we defined one enum in dart, and the matching enum in Android. To make sense of the int passing through channel all you need to do is recast to the respective enum.

/// adds a new mode that is not recognized by the `SensitiveContent` widget.
///
/// This mode cannot be used to set the sensitivity level of a `SensitiveContent` widget.
_unknown,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for cases where you send _unknown through channel, can we just return error instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reidbaker and I discussed this at length. I'm sure we could accomplish the same thing that way, as I mentioned before, but this gives a standardized way for any platform to signal that an unknown value was detected and we should gracefully fail and not crash the app.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

throw FlutterError('Failed to retrieve content sensitivity: $e');
}

final ContentSensitivity contentSensitivity = ContentSensitivity.values.firstWhere(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we use int, we don't need to do a linear look up and can just recast with ContentSensitivity.values[result]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sold on converting back to ints. See #158473 (comment).

}

// Re-register SensitiveContent widget if the sensitivity changed.
_sensitiveContentRegistrationFuture = SensitiveContentHost.register(widget.sensitivity);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should have one method to replace the new old registration with new one instead of two. having two separate setSenstiveContent call worries me a bit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this way, you only need to keep track of one future

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually did this before but I thought it was breaking something. It was not; fixed.


/// Host of the current content sensitivity level for the widget tree that contains
/// some number [SensitiveContent] widgets.
class SensitiveContentHost {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some where in the doc should mention this is not ready for production

Copy link
Contributor

@justinmc justinmc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with nits, assuming the checks pass 👍

);
} catch (e) {
// Content sensitivity failed to be set.
throw FlutterError('Content sensitivity $contentSensitivity failed to be set: $e');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: What's the benefit of catching and rethrowing here?

Copy link
Contributor Author

@camsim99 camsim99 Mar 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Undid this.; now the SensitiveContentHost just catches the PlatformException

if (defaultTargetPlatform != TargetPlatform.android) {
return false;
}
return (await sensitiveContentChannel.invokeMethod<bool>('SensitiveContent.isSupported'))!;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the ! here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method channel call results in Future<bool> which according to the analyzer, can be null.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh right. This is probably another thing that we should do error handling for. Currently it looks like there is no try/catch where this is called in _register. Currently if this call fails, or if it unexpectedly returns null, then the app will crash. Instead I guess we should just consider it to be unsupported?

'when SensitiveContentService.setContentSensitivity fails, SensitiveContentHost.register throws FlutterError but still updates calculatedContentSensitivity',
(WidgetTester tester) async {
int setContentSensitivityCall = 0;
TestDefaultBinaryMessengerBinding.instance.defaultBinaryMessenger.setMockMethodCallHandler(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think after doing this you also should remove it:

      addTearDown(
        () => tester.binding.defaultBinaryMessenger.setMockMethodCallHandler(
          SystemChannels.sensitiveContent,
          null,
        ),
      );

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't the tearDown do this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah sorry I totally missed that above, this is good as-is 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for splitting the tests into different files like this.

Comment on lines +116 to +118
// Two pumps to complete re-registration, then re-build SensitiveContent widget.
await tester.pump();
await tester.pump();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explaining this. I'm glad you ended up not needing pumpAndSettle or expectLater.

@camsim99
Copy link
Contributor Author

My understanding from the reviews and an offline discussion earlier was that this is ready to land. There's some cleanup and significant comments here, so I'm not sure I agree anymore.

I'll be OOO starting midday tomorrow, so I will try to address these comments, but will likely not land it. @reidbaker if I get it to a place where I feel confident that the reviews are addressed, I'll let you know in case you can land this while I'm out. Thank you everyone for the feedback and helping push this forward!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
engine flutter/engine repository. See also e: labels. framework flutter/packages/flutter repository. See also f: labels. platform-android Android applications specifically
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants