[TrimmableTypeMap] Add TypeMap assembly and JCW Java source generators#10808
Draft
simonrozsival wants to merge 30 commits intodev/simonrozsival/trimmable-typemap-02-scannerfrom
Draft
Conversation
627fef2 to
408d73d
Compare
ba936c9 to
61ac7ed
Compare
ddddd2f to
83a50be
Compare
Introduce Microsoft.Android.Build.TypeMap assembly with a two-phase scanner that extracts Java peer type information from .NET assemblies using SRM (no Mono.Cecil dependency). Key components: - JavaPeerScanner: scans assemblies for [Register], [Export], and component attributes to build JavaPeerInfo entries - AssemblyIndex: per-assembly O(1) lookup index built in phase 1 - JavaPeerInfo/MarshalMethodInfo: rich data model for downstream generators (PR2/PR3) - CRC64 package name computation via System.IO.Hashing Tests: - 62 xUnit tests covering all scanner code paths - 3 integration tests comparing side-by-side with legacy Cecil-based scanner on Mono.Android.dll (all pass)
- Replace hardcoded version paths with dynamic directory scanning - Add ScannerDiagnostics_MonoAndroid test with sanity checks (>3000 types, >500 interfaces, >10000 marshal methods) - Fix duplicate key issue in comparison tests using GroupBy - Reference pre-built DLLs instead of building from source
Single ExactTypeMap_MonoAndroid test verifies every JNI → managed type entry matches exactly between legacy and new scanners: no missing, no extra, no managed name or skip flag mismatches.
…ison test - Remove ParseConnectorString, CallbackTypeName, CallbackAssemblyName from scanner data model (connector parsing deferred to PR2) - Add ExactMarshalMethods_MonoAndroid integration test comparing method registrations between legacy Cecil and new SRM scanner - Legacy extraction reads [Register] from both methods and properties - ~55K methods compared across ~6K types with <0.2% difference (known gaps in generic type handling and interface property connectors)
…ison test Reworked ExactMarshalMethods_MonoAndroid to compare the full set of managed types per JNI name (aliases), instead of picking a single representative type. This eliminates all 148 false-positive differences that were caused by duplicate JNI name collisions. Key changes: - New TypeMethodGroup record: groups methods by managed type name - RunLegacyScanner returns Dict<jniName, List<TypeMethodGroup>> - RunNewScanner returns the same structure - Filter Invoker types (DoNotGenerateAcw + name ends with 'Invoker') from the legacy side to match our scanner's intentional exclusion - Include types from dataSets.JavaToManaged that aren't in javaTypes - Normalize Cecil's '/' nested type separator to '+' (CLR format) Result: exact match — 7072 type groups, 7040 JNI names, 55569 methods, zero differences between legacy Cecil scanner and new SRM scanner.
…ion tests Replace 9 <Reference> items with ProjectReferences: - Xamarin.Android.Build.Tasks (transitive deps cover Cecil, JCW, etc.) - Mono.Android (ReferenceOutputAssembly=false; built as dependency for ref assembly) Discover Mono.Android.dll path via MSBuild AssemblyMetadata target instead of runtime directory scanning. Remove SkippableFact dependency. Switch TFM to $(DotNetTargetFramework) (net11.0) to match Mono.Android.
New integration tests comparing legacy Cecil scanner vs new SRM scanner
on Mono.Android.dll (~7000 types):
- ExactBaseJavaNames: verifies base Java type names match
- ExactImplementedInterfaces: verifies interface JNI names match
- ExactActivationCtors: verifies activation ctor presence, declaring
type, and style (XI vs JI) match
- ExactJavaConstructors: verifies [Register("<init>",...)] ctors match
- ExactTypeFlags: verifies IsInterface, IsAbstract, IsGenericDefinition,
DoNotGenerateAcw flags match
Scanner fixes:
- Fix TypeSpecification (generic base type) resolution: strip generic
args from decoded signature to find the generic definition in
TypesByFullName (e.g. "Ns.Type`1<A>" → "Ns.Type`1")
- Scan Java.Interop.dll alongside Mono.Android.dll for cross-assembly
base type resolution (JavaException → java/lang/Throwable)
Test infrastructure:
- Replace AssemblyMetadata approach with direct <Reference> to
Mono.Android ref assembly + nameof(Java.Lang.Object) compile-time check
- Add AllAssemblyPaths helper that includes Java.Interop.dll
- Filter new scanner results to primary assembly only
Invoker types (DoNotGenerateAcw=true, name ends with Invoker) are now included in the scanner's output. They will be filtered later by downstream generators. This matches the design principle that the scanner should discover ALL Java peer types. - Remove IsInvokerType skip from JavaPeerScanner - Remove IsCecilInvokerType filter from integration tests - Update unit tests: invokers are now expected in scan results - Add FindByManagedName helper for disambiguating shared JNI names - Handle invoker/self-reference base name differences in comparison
- JavaPeerInfo and related types: replace constructors with { get; set; }
properties and object initializer syntax
- Remove // ==== separator comments from tests and fixtures (-300 lines)
- Rename project from Microsoft.Android.Build.TypeMap to Microsoft.Android.Sdk.TrimmableTypeMap to match naming of Microsoft.Android.Sdk.ILLink and Microsoft.Android.Sdk.Analysis - Fix ParseJniType crash on malformed JNI signatures - Add cycle detection to ExtendsJavaPeer - Fix SignatureTypeProvider nested type resolution - Implement GetUnderlyingEnumType via value__ field inspection - Implement TryGetTypeProperty for BackupAgent and ManageSpaceActivity - Fix InternalsVisibleTo: remove AssemblyName hack, add proper entries - Add 7 ParseJniSignature unit tests
Implements generators for #10799: - JcwJavaSourceGenerator: generates .java files for ACW types from JavaPeerInfo - TypeMapModelBuilder: transforms JavaPeerInfo → TypeMapAssemblyModel (IR/AST) - TypeMapAssemblyEmitter: mechanical SRM-based PE emitter from IR model - TypeMapAssemblyGenerator: high-level API composing builder + emitter - JniSignatureHelper: JNI signature parsing and CLR type encoding Key design: - 1 typemap assembly per input assembly for better caching - IR model separates 'what to generate' from 'how to serialize to IL' - Model builder tests are the primary unit tests (148 total, all passing) Generated assemblies contain: - [assembly: TypeMap] attributes per JNI type - Proxy types (JavaPeerProxy subclasses) with CreateInstance, TargetType - [UnmanagedCallersOnly] UCO wrappers for marshal methods/constructors - RegisterNatives with function pointer registration - IgnoresAccessChecksToAttribute for cross-assembly calls
- TypeMapAssemblyModel → TypeMapAssemblyData - TypeMapEntryModel → TypeMapAttributeData - ProxyTypeModel → JavaPeerProxyData - TypeRefModel → TypeRefData - UcoMethodModel → UcoMethodData - UcoConstructorModel → UcoConstructorData - NativeRegistrationModel → NativeRegistrationData - TypeMapModelBuilder → ModelBuilder
30 new tests covering every test fixture type: - MCW types: Object, Activity, Throwable, Exception, Service, Context, View, Button - User ACW types: MainActivity, MyHelper, TouchHandler, CustomView, AbstractBase - Interface types: IOnClickListener (with invoker dedup) - Nested types: Outer$Inner, ICallback$Result proxy naming - Multi-interface: ClickableView, MultiInterfaceView - Export methods: ExportExample - Generic types: GenericHolder - Full pipeline tests: scan → model → emit → read back PE Validates UCO wrapper signatures for all JNI types (bool, int, float, long, double, object, array), constructor wrappers, native registrations, TypeMap attribute counts, and proxy type names in emitted assemblies. 178 tests pass, 1 skipped.
…erator
Critical changes for TrimmableTypeMap:
- TypeMapAttributeData now supports 2-arg (unconditional) and 3-arg (trimmable):
- 2-arg: ACW user types (Android can instantiate), essential runtime types
(java/lang/Object, Throwable, Exception, etc.)
- 3-arg: MCW bindings and interfaces — trimmer preserves proxy only if
target type is referenced by the app
- Alias detection: when multiple .NET types share the same JNI name,
they get indexed entries ("jni/name", "jni/name[1]", "jni/name[2]")
with distinct proxy types
- RootTypeMapAssemblyGenerator: generates _Microsoft.Android.TypeMaps.dll
with [assembly: TypeMapAssemblyTarget("name")] for each per-assembly
typemap assembly
208 tests pass, 1 skipped.
…overage Review-driven fixes: Bug fixes: - Fix critical bug: constructor JNI signatures were hardcoded to "()V" in BuildNativeRegistrations. Now propagated from JavaConstructorInfo through UcoConstructorData.JniSignature to NativeRegistrationData. - Fix non-deterministic alias ordering: replaced Dictionary with SortedDictionary, sort alias peers by ManagedTypeName. - Fix Export attribute ThrownNames parsing: string[] was not being decoded from ImmutableArray<CustomAttributeTypedArgument<string>>. Test improvements: - Cache scanner results with static Lazy<> in all test classes - Add parameterized constructor JNI signature test - Add fixture-based CustomView constructor signature assertions - Add PE blob validation tests (2-arg vs 3-arg TypeMap attributes) - Add determinism test (same input → same output) - Add Export with throws clause test fixture + JCW test - Fix Build_CreatesOneEntryPerPeer for alphabetical ordering Code quality: - Add ECMA-335 comment explaining Type args as serialized strings - Add comment explaining UCO constructor 2-param signature - Fix doc comment: remove 'Intermediate representation' wording 215 tests pass, 1 skipped.
…uted IgnoresAccessChecksTo, edge-case tests
…exclusion, invoker filtering, managed-name proxy naming, root assembly cleanup - Make ModelBuilder a static class with Build() method - Add IsInvokerType() to filter invokers from alias grouping (get proxy, no TypeMap entry) - Add IsImplementorOrEventDispatcher() to exclude from unconditional entries - Change proxy naming from JNI-based to managed-name-based for uniqueness - Remove unused stringRef/systemRuntimeInteropServicesRef from RootTypeMapAssemblyGenerator - Add Implementor and EventDispatcher test fixtures - Add tests for Implementor/EventDispatcher trimmability - Add root assembly attribute blob value verification test - All 225 tests pass
…ame heuristics, add tests - Clear _typeRefCache in Emit() alongside _asmRefCache to prevent stale handles on reuse - Extract MonoAndroidPublicKeyToken and SystemRuntimeVersion as named constants - Document name-based IsInvokerType/IsImplementorOrEventDispatcher heuristic limitations - Fix stale doc comment on JavaPeerProxyData.TypeName - Add FullPipeline_Mixed2ArgAnd3Arg_BothSurviveRoundTrip test - Add FullPipeline_CustomView_UcoConstructorHasExactlyTwoParams PE-level test - Add FullPipeline_GenericHolder_ProducesValidAssembly PE pipeline test - Add edge case tests documenting name-based detection limitations - Refactor ReadFirstTypeMapAttributeBlob into ReadAllTypeMapAttributeBlobs - All 230 tests pass
…support - Invoker types no longer get their own proxy types or TypeMap entries. They are only referenced as a TypeRef in the interface proxy's get_InvokerType property and CreateInstance method. - Invoker detection now uses explicit relationship from [Register] third argument (InvokerTypeName) instead of name-based heuristic. - Interface proxy CreateInstance creates the invoker type, not the interface itself. - Added dotnetVersion parameter to TypeMapAssemblyEmitter, TypeMapAssemblyGenerator, and RootTypeMapAssemblyGenerator constructors (will be passed from $(DotNetTargetVersion) MSBuild property later). - JCW generator now uses SuperArgumentsString for [Export] constructor super() calls instead of always forwarding all parameters. - Documented PE-reading test helpers with clear comments explaining the approach and its limitations.
…Attribute<Java.Lang.Object> Instead of defining a non-generic TypeMapAssemblyTargetAttribute in the root assembly, reference the existing generic TypeMapAssemblyTargetAttribute`1 from System.Runtime.InteropServices and close it with Java.Lang.Object as the type argument. This matches the runtime API: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.typemapassemblytargetattribute-1
Each '// ---- Section ----' comment block is now a nested class inside the outer test class. This allows running individual test groups in isolation and makes test failures easier to locate. Shared helpers remain as static methods on the outer class.
After rebasing onto the renamed base branch, Generator source and test files landed under the old Microsoft.Android.Build.TypeMap paths. Move them to the correct Microsoft.Android.Sdk.TrimmableTypeMap directories and update all namespace references.
…ciation, fix IgnoresAccessChecksTo - Replace CreateManagedPeer with spec-compliant CreateInstance: - Leaf ctor: newobj T::.ctor(IntPtr, JniHandleOwnership) - Inherited ctor: GetUninitializedObject + call Base::.ctor - Interface: newobj TInvoker::.ctor(IntPtr, JniHandleOwnership) - Generic: throw new NotSupportedException() - Add TypeMapAssociationAttribute emission for alias groups - Include base ctor assembly in IgnoresAccessChecksTo - Add ActivationCtorData and IsGenericDefinition to proxy model - Add 5 new tests covering all CreateInstance paths, TypeMapAssociation, and IgnoresAccessChecksTo for inherited ctors
- Make HasActivation a computed property (derived from ActivationCtor/InvokerType) - Merge EmitSinglePeer into EmitPeers (handles both single and alias cases) - Extract EmitCreateInstanceBody to deduplicate method signature across 5 paths - Make BuildEntry's jniName parameter required (no longer optional)
- Remove ImplementsIAndroidCallableWrapper (was always == IsAcw) - Collapse EmitTypeMapAttribute 2-arg/3-arg into shared blob writing - Extract duplicated UCO signature lambda in EmitUcoMethod - Simplify invoker filtering with LINQ - Extract AddIfCrossAssembly helper for IgnoresAccessChecksTo
- Reuse 3 BlobBuilder instances (_sigBlob, _codeBlob, _attrBlob) instead of allocating a new one per method body, attribute, and member ref. - Pre-compute the UCO attribute BlobHandle (always same 4 bytes). - Use (string, string) tuple for type ref cache key instead of string interpolation — avoids allocation on every cache lookup.
Add JniParameterInfo, JavaConstructorInfo, and additional properties to MarshalMethodInfo and JavaPeerInfo that generators require (JniReturnType, NativeCallbackName, Parameters, JavaConstructors, ManagedTypeNamespace, ManagedTypeShortName). Extend JniSignatureHelper with raw type string parsing. Enrich scanner output with these derived fields.
The UCO constructor wrapper signature must match the JNI native method signature (jnienv + self + constructor params) for correct ABI. Previously it only accepted (IntPtr, IntPtr), causing a calling convention mismatch when JNI dispatches constructors with parameters. The body still calls ActivateInstance(self, typeof(T)) — the constructor params are consumed but not forwarded, since peer activation uses the (IntPtr, JniHandleOwnership) activation ctor, not the user constructor.
83a50be to
4e7d06c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #10789
Stacked on #10805
Summary
This PR adds the TypeMap assembly generators and JCW Java source generators on top of the scanner from PR1. Together, these components form the complete pipeline: scan assemblies → build model → emit TypeMap assemblies + JCW
.javafiles.What's included
TypeMap Assembly Generator (
TypeMapAssemblyEmitter+ModelBuilder)TypeMapAttribute<Java.Lang.Object>(2-arg unconditional, 3-arg trimmable) with proxy self-application pattern_Microsoft.Android.TypeMaps.dll): Central assembly withTypeMapAssemblyTargetAttribute<Java.Lang.Object>pointing to all per-assembly TypeMap DLLsCreateInstancemethod: Generates correct IL for 5 activation patterns:return nullthrow new NotSupportedExceptionnewobj TInvoker::.ctor(IntPtr, JniHandleOwnership)newobj T::.ctor(IntPtr, JniHandleOwnership)GetUninitializedObject+castclass+call Base::.ctor[UnmanagedCallersOnly]wrappers: UCO-attributed static methods wrapping each marshal method forRegisterNatives[IgnoresAccessChecksTo]: Emitted for all cross-assembly references (including base ctor assemblies)TypeMapAssociationAttribute: Emitted for alias types pointing to their canonical peerJCW Java Source Generator (
JcwJavaSourceGenerator).javafiles withregisterNativescalls instatic {}blocksTextWriter-based output (no intermediate string allocations)super()callsthrowsclauses fromThrownNamesScanner Enrichment (fix-up for generators)
JniParameterInfo/JavaConstructorInfo— structured JNI parameter data for JCW constructor generationParseJniParameters/BuildJavaConstructors— enrichment methods on scanner outputJniSignatureHelper—ParseParameterTypeStrings,ParseReturnTypeStringfor JNI signature decompositionPerformance Optimizations
BlobBuilderfields withClear(), pre-computed UCO attribute blob,(string, string)tuple cache key for type refsGetStringcalls, tuple cache key forextendsJavaPeerCache, cleanStringBuilderhex formattingTests
Follow-up