Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add experimental data assets from hooks #164094

Open
wants to merge 33 commits into
base: master
Choose a base branch
from

Conversation

mosuem
Copy link
Contributor

@mosuem mosuem commented Feb 25, 2025

Rebase of #159675

This PR adds bundling support for the experimental dart data asset feature: Dart packages with hooks can now emit data assets which the flutter tool will bundle.

It relies on flutter's existing asset bundling mechanism (e.g. entries in AssetManifest.json, DevFS syncing in reload/restart, ...).

The support is added under an experimental flag (similar to the existing native assets experimental flag).

Also, kNativeAssets is removed to also bundle data assets on flutter build bundle.

cc @bkonyi

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@github-actions github-actions bot added tool Affects the "flutter" command-line tool. See also t: labels. team-ios Owned by iOS platform team labels Feb 25, 2025
@mosuem mosuem force-pushed the rebaseMartinsDataAssets branch from 5e36ce4 to 9a2f7e6 Compare February 25, 2025 17:19
@mosuem mosuem marked this pull request as ready for review February 27, 2025 14:40
@mosuem mosuem changed the title Add experimental data assets Add experimental data assets from hooks Feb 27, 2025
@mosuem
Copy link
Contributor Author

mosuem commented Feb 28, 2025

There seems to be a test failure related to the ordering of stdouts at https://github.com/mosuem/flutter/blob/rebaseMartinsDataAssets/packages/flutter_tools/test/integration.shard/overall_experience_test.dart#L123 - not sure if this is intentional or not.

@mosuem mosuem force-pushed the rebaseMartinsDataAssets branch 2 times, most recently from 840b811 to f71c5e9 Compare March 4, 2025 08:39
@github-actions github-actions bot added the a: desktop Running on desktop label Mar 4, 2025
@mkustermann
Copy link
Member

mkustermann commented Mar 5, 2025

@mosuem You should be able to revert the changes to packages/flutter_tools/test/integration.shard/transition_test_utils.dart: I've pulled the refactoring of the test framework into a different CL that already landed on the main branch (the change landed as a44f745)

@mkustermann
Copy link
Member

There seems to be a test failure related to the ordering of stdouts at https://github.com/mosuem/flutter/blob/rebaseMartinsDataAssets/packages/flutter_tools/test/integration.shard/overall_experience_test.dart#L123 - not sure if this is intentional or not.

The ordering of the patterns in

          Multiple(
            <Pattern>[
              RegExp(
                r'^Reloaded 0 libraries in [0-9]+ms \(compile: \d+ ms, reload: \d+ ms, reassemble: \d+ ms\)\.$',
              ),
              'called reassemble',
              'called paint',
            ],
            handler: (String line) {
              processManager.killPid(pid, ProcessSignal.sigusr2);
              return null;
            },
          ),

is not relevant. You can see how Multiple is implemented. It just waits for all 3 patterns to occur (in any order) and only procceedes once all patterns have been found. So if it hangs in this step, it's a sign that one of the patterns wasn't found.

@mkustermann
Copy link
Member

There seems to be a test failure related to the ordering of stdouts at https://github.com/mosuem/flutter/blob/rebaseMartinsDataAssets/packages/flutter_tools/test/integration.shard/overall_experience_test.dart#L123 - not sure if this is intentional or not.

The root cause I believe is the modified code (which should be reverted - as mentioned earlier):

diff --git a/packages/flutter_tools/test/integration.shard/transition_test_utils.dart b/packages/flutter_tools/test/integration.shard/transition_test_utils.dart
index 73f219ac83..39210532a4 100644
--- a/packages/flutter_tools/test/integration.shard/transition_test_utils.dart
+++ b/packages/flutter_tools/test/integration.shard/transition_test_utils.dart
   @protected
-  bool lineMatchesPattern(String line, Pattern pattern, bool contains) {
-    if (pattern is RegExp) {
-      // Ideally this would also distinguish between "contains" and "equals"
-      // operation.
-      return line.contains(pattern);
+  bool lineMatchesPattern(String line, Pattern pattern) {
+    if (pattern is String) {
+      return contains ? line.contains(pattern) : line == pattern;
     }
     return contains ? line.contains(pattern) : line == pattern; // XXX
   }

This XXX is incorrect. The line == pattern will never work because line is a String and pattern is a RegExp.

final _Asset asset = _Asset(
baseDir: baseDir,
relativeUri: relativeUri,
entryUri: Uri.parse('packages/${dataAsset.package}/${dataAsset.name}'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just confirming, has this been tested on Windows?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The windows test was failing - I applied for the cloudtop permissions and can now finally debug it!

@mosuem mosuem force-pushed the rebaseMartinsDataAssets branch from 7721ce5 to 5d27153 Compare March 18, 2025 17:52
@dcharkes dcharkes self-requested a review March 19, 2025 14:15
required Map<String, String> environmentDefines,
required FlutterNativeAssetsBuildRunner buildRunner,
required List<Architecture> architectures,
required Uri projectUri,
required FileSystem fileSystem,
required OS? targetOS,
required bool linkingEnabled,
required bool codeAssetSupport,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we want to run this method with two modes after this PR:

  • CodeAssets + DataAssets
  • DataAssets (web only)

It seems weird that we can have codeAssetSupport: false but we still need to supply an OS and Architectures

I suggest packaging the codeAssetSupport (with architectures and os), dataAssetSupport and isWeb in objects.

sealed class FlutterAssetConfig {}

class FlutterDataAssetConfig extends FlutterAssetConfig {}

class FlutterCodeAssetConfig extends FlutterAssetConfig {
  FlutterCodeAssetConfig(this.architectures, this.os);

  final List<Architecture> architectures;
  final OS os;
}

Since we seem to run this method with multiple asset types at the same time (and then invoke the hooks for the asset types separately in this method), the param needs to be as follows:

Future<DartHookResult> _runDartHooks({
  required List<FlutterAssetConfig> assetConfigurations, // or a Set, but that adds hashing
  // ...

@@ -400,8 +414,39 @@ class ManifestAssetBundle implements AssetBundle {
}
}

for (final DataAsset dataAsset in dartHookResult?.dataAssets ?? <DataAsset>[]) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not in the isolated directory, so we can't use DataAssets here. The HookResult should contain something else than DataAsset and CodeAsset, it should return the info in a format not requiring to import package:native_assets_cli.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we can't use package:native_assets_cli outside of isolated?

Copy link
Contributor

@dcharkes dcharkes Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrrect, you can't use it here. We'll have to extract the info you need from DataAssets inside isolated/ and then make that info available here.

root.path,
<Transition>[
Barrier.contains(
'Conflicting assets: The asset "asset: packages/data_asset_dependency/id1.txt" was declared in the pubspec and the hook',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can logic such as this be tested on the general shard (unit tests that mock running the hooks)? That requires less resources to run.

See test/general.shard/isolated/native_assets_test.dart with buildRunner: FakeFlutterNativeAssetsBuildRunner(.

I'll leave the balance between mocked-out-tests and full integration tests is something for Ben to decide on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a: desktop Running on desktop team-ios Owned by iOS platform team tool Affects the "flutter" command-line tool. See also t: labels.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants