DeveloperLand.com [Managed and Unmanaged Code Integration]

The Common Language Infrastructure [^] (CLI) is designed to make it "easy" to interoperate with existing code. In principal, all you need to do is create a DllImport [^] function declaration for the existing code to invoke, and the runtime will handle the rest. For example:

[DllImport ("libc.so")]
private static extern int getpid ();

Please note that most of the classes and enumerations mentioned in this document reside in the System.Runtime.InteropServices [^] namespace.

The above C# function declaration would invoke the POSIX getpid(2) system call on platforms that have the libc.so library (other platforms would generate a MissingMethodException [^]). Simple. Straightforward. What could be easier?

There are three problems with this: (1) specifying the library in the DllImport statement; (2) determining what function to actually invoke. (3) most existing code is far more complex. Strings will need to be passed, structures may need to be passed, memory management practices will become involved... Existing code is a complex beast, and the interop layer needs to support this complexity.

Top

Library Handling

How does the runtime find the library specified in the DllImport attribute? This question is inherently platform specific.

Top

Windows DLL Handling

From the MSDN LoadLibrary documentation, the DLLs needed by the program are searched for in the following order:

The directory from which the application loaded.
The current directory
The system directory. Use the GetSystemDirectory() function to get the path of this directory.
The 16-bit system directory.
The Windows directory. Use the GetWindowsDirectory() function to get the path of this directory.
The directories that are listed in the PATH environment variable.

Of course, reality isn't quite that simple. In practice, the "system" directory is actually %WINDIR%\system32, except on Windows 9x platforms where it's %WINDIR%\system. The 16-bit system directory is typically %WINDIR%\system, but isn't recognized as a separate search directory on Windows 9x platforms.

Furthermore, on Windows Server 2003 and Windows XP SP1, the registry entry HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SafeDllSearchMode alters the above ordering. If this is 1 (the default), then the current directory is searched after the system and Windows directories. This is a security feature (it prevents a trojan library from being loaded instead of, for example, OLE32.DLL), but it turns the above list into: 1, 3, 4, 5, 2, 6.

Top

Linux Shared Library Handling

From the dlopen(3) man page, the necessary shared libraries needed by the program are searched for in the following order:

A colon-separated list of directories in the user's LD_LIBRARY_PATH environment variable. This is a frequently-used way to allow native shared libraries to be found by a CLI program.
The list of libraries cached in /etc/ld.so.cache. /etc/ld.so.cache is created by editing /etc/ld.so.conf and running ldconfig(8). Editing /etc/ld.so.cache is the preferred way to search additional directories, as opposed to using LD_LIBRARY_PATH, as this is more secure (it's more difficult to get a trojan library into /etc/ld.so.cache than it is to insert it into LD_LIBRARY_PATH).
/lib, followed by /usr/lib.

Top

Mac OS X Framework and .dylib Handling

The Framework and library search path is:

A colon-separated list of directories in the user's DYLD_FRAMEWORK_PATH environment variable.
A colon-separated list of directories in the user's DYLD_LIBRARY_PATH environment variable.
A colon-separated list of directories in the user's DYLD_FALLBACK_FRAMEWORK_PATH environment variable, which defaults to the directories:
- ~/Library/Frameworks
- /Library/Frameworks
- /Network/Library/Frameworks
- /System/Library/Frameworks
A colon-separated list of directories in the user's DYLD_FALLBACK_LIBRARY_PATH environment variable, which defaults to the directories:
- ~/lib
- /usr/local/lib
- /lib
- /usr/lib

TODO: Will mono support both frameworks and dylibs?

Top

Library Names

Knowing where to look for the library is only half of the problem. Knowing what library to load is the other half.

Different platforms have different naming conventions. Windows platforms append .DLL to the library name, such as OLE32.DLL. Linux platforms use a lib prefix and a .so suffix ¹ [^]. Mac OS X platforms have a lib prefix and a .dylib suffix, unless they're a Framework, in which case they're a directory and things get more complicated.

Note 1: Strictly speaking, Unix shared libraries are typically versioned, and the version number follows the .so suffix. For example, libfreetype.so.6.3.3 is a fully versioned library. Versioning throws a "wrench" into the works, and is best dealt with through Mono's <dllmap/> mechanism; see below for details.

If you have control over the library name, keep the above naming conventions in mind and don't use a platform-specific library name in the DllImport statement. Instead, just use the library name itself, without any prefixes or suffixes, and rely on the runtime to find the appropriate library at runtime. For example:

[DllImport ("MyLibrary")]
private static extern void Frobnicate ();

Then, you just need to provide MyLibrary.dll for Windows platforms, libMyLibrary.so for Unix platforms, and libMyLibrary.dylib for Mac OS X platforms.

What if you don't have the same name across all platforms? For example, the GTK+ library name on Windows is libgtk-win32-2.0-0.dll, while the Unix equivalent library is libgtk-x11-2.0.so. How do you write portable Platform Invoke (P/Invoke) code that will work cross-platform?

The short answer is that you don't. There is no standard way of specifying platform-specific library names.

However, as an extension, Mono provides a library mapping mechanism. Two places are searched for library mappings: in the $prefix/etc/mono/config XML file, and in a per-assembly .config file, located in the same directory as the assembly. These files contains <dllmap/> elements, which map an input library (the library specified in the DllImport statement) to the actual platform-specific library to load. For example:

   <configuration>
      <dllmap dll="libgtk-win32-2.0-0.dll" 
         target="libgtk-x11-2.0.so" />
   </configuration>

Unlike .NET, Mono permits .DLL assemblies to have .config files, which are only used for this library mapping mechanism.

Using this mechanism, the Mono-endorsed way of specifying DllImport library names is to always use the Windows library name (as Microsoft .NET has no library mapping mechanism), and then provide a mapping in the per-assembly .config file. This is what the Gtk# library does.

This mechanism can also be used to load strongly-versioned libraries on Unix platforms. For example:

   <configuration>
      <dllmap dll="gtkhtml-3.0" 
         target="libgtkhtml-3.0.so.4" />
   </configuration>

Top

Invoking Unmanaged Code

As far as managed code is concerned, unmanaged code is invoked merely by invoking a method with an associated DllImport attribute. The CLI runtime must do more work to actually invoke the unmanaged code.

In principal, this is a straightforward process. The library specified in the DllImport attribute is loaded, as described above. Then, the specified function is looked up (via GetProcAddess() or dlsym(3)). Finally, the function is invoked.

But what string is used for the function lookup (in GetProcAddress() or dlopen(3))? By default, the name of the managed code method is used, which is why getpid() in the above example [^] invokes getpid(2) from the C library.

Alternatively, the DllImport attribute's EntryPoint [^] field can be set, and that string will be used instead.

Either way, the string used is assumed to refer to a C ABI-compatible function exported by the specified library. On some platforms, this may cause a leading underscore to be prefixed to the symbol name. Other platforms generate no mangling.

Note that a C ABI is assumed. This makes it nearly impossible to directly invoke functions that are not C ABI compatible, such as C++ library functions that are not extern "C". Some variation on the C ABI is permitted, such as variation in the function's CallingConvention [^]. The default CallingConvention is platform-specific. Under Windows, StdCall [^] is the default, as this is used for most Win32 API functions. Under Unix platforms, Cdecl [^] is the default.

Calling convention can be specified in C code by using the __stdcall and __cdecl compiler intrinsics under Microsoft Visual C++, and by using the __attribute__((stdcall)) and __attribute__((cdecl)) compiler intrinsics under GCC.

Does having the default CallingConvention vary between platforms cause portability problems? Yes. All the more reason to write as much code as possible as managed code, avoiding the whole P/Invoke/marshaling conundrum in the first place.

If you need to invoke C++ code, you have two choices: (1) make the C++ function extern "C", treat it as a C function, and make sure that it uses a known calling convention; (2) don't make the function extern "C", but make sure it uses a known calling convention. If you use option (2), you'll need to set the DllImport.EntryPoint field to the C++ mangled function name, such as _Z6getpidv. You can retrieve the mangled name through your compiler's binary tools, such as OBJDUMP.EXE or nm(1). Note that C++ mangled names are highly compiler specific, and will (a) make your .NET assembly platform specific (you'll need a different assembly for each different platform); (b) require updating the .NET assembly every time you change C++ compilers (as the C++ name mangling scheme can -- and frequently will -- change); and (c) be really ugly to maintain because of (a) and (b). Option (2) is not recommended.

Top

Exception Propagation

The above section mentioned a key point: P/Invoke assumes that the unmanaged code conforms to the C ABI. C doesn't support exceptions. As such, it is assumed that runtime exceptions will not propagate through unmanaged code.

Furthermore, it's fairly simple for an exception to propagate through unmanaged code whenever unmanaged code invokes managed code. This typically occurs through the use of callbacks -- using a function pointer on the unmanaged side which can invoke a delegate on the managed side. It is very important that the unmanaged code not propagate any exceptions -- it must catch all exceptions, or else the unmanaged code calling the delegate will break.

The problem is, again, C doesn't support exceptions. C++ supports exceptions, BUT, and this is crucial, the C++ exception mechanism will be (with one exception) different from the managed code exception mechanism. Since managed code doesn't know about unmanaged code's exception handling support (C is assumed, and C doesn't support exceptions; I'm sounding like a broken record), unmanaged exception handling support might as well not exist, because it won't be used.

Given the above scenario -- unmanaged code invokes function pointer which generates a managed exception -- what would happen? The managed exception handling mechanism is executed: the stack is searched for an appropriate exception handler, then the stack is unwound, with any finally blocks executed during the stack unwind process.

Note two things. Managed code will be walking the stack, requiring that the CPU Stack Pointer and Instruction Pointers be set. Consequently, unmanaged code cannot participate in stack unwinding, as it will never be notified that a stack unwind is occurring.

Think about that for a minute. If alarms are not sounding in your head, you're in deep, deep trouble. Consider this unmanaged C code:

   typedef void (*Handler) (const char *message);
   void InvokeHandler (Handler handler)
   {
      char *message = malloc (10);
      strcpy (message, "A Message");
      (*handler)(message);
      free (message);
   }

If handler is a pointer to a managed delegate which may throw an exception, then free(3) will not be executed, resulting in a memory leak. C++ destructors won't help you either, as destructors still require the execution of some code, and that code will never be invoked, as it's not C++ which is unwinding the stack, but managed code, which doesn't know about C++ exception handling.

Obviously, the flip-side of this scenario -- a C++ exception being propagated into managed code -- is equally bad. As long as managed and unmanaged code use different exception handling mechanisms, exceptions can not be mixed between them.

The moral of this story: don't let exceptions propagate between managed and unmanaged code. The results won't be pretty.

This is particularly pertinent when wrapping C++ methods. C++ exceptions will need to be mapped into an "out" parameter or a return value, so that managed code can know what error occurred, and (optionally) throw a managed exception to "propagate" the original C++ exception.

The one exception to this is when you use both .NET and Microsoft Visual C++ to compile the unmanaged code. .NET uses Windows Structured Exception Handling (SEH) at the P/Invoke layer for its exception handling mechanism, and Microsoft Visual C++ uses SEH to implement C++ exception handling and supports the use of SEH in C as a language extension through the __try, __except, and __finally keywords. SEH is a Microsoft extension; it does not exist outside of Microsoft and .NET, and as such is not portable.

Top

Marshaling

How does code interop work? Given a managed call site (the function call), and an unmanaged callee site (the function that's being called), each parameter in the call site is "marshaled" (converted) into an unmanaged equivalent. The marshaled data is in turn placed on the runtime stack (along with other data), and the unmanaged function is invoked.

The complexity is due to the marshaling. For simple types, such as integers and floating-point numbers, marshaling is a bitwise-copy ("blitting"), just as would be the case for unmanaged code. In some cases, marshaling can be avoided, such as when passing structures by reference to unmanaged code (a pointer to the structure is copied instead). It's also possible to obtain more control over marshaling, using custom marshaling [^].

String types introduce additional complexity, as you need to specify the form of string conversion. The runtime stores strings as UTF-16-encoded strings, and these will likely need to be marshaled to a more appropriate form (ANSI strings, UTF-8 encoded strings, etc.). Strings get special support.

Some marshaling behavior is controlled through the DllImport [^] and MarshalAs [^] attributes.

Top

Memory Boundaries

Managed and unmanaged memory should be considered to be completely separate. Managed memory is typically memory allocated on a garbage-collected heap, while unmanaged memory is anything else: the ANSI C memory pool allocated through malloc(3), custom memory pools, and garbage-allocated heaps outside the control of the CLI implementation (such as a LISP or Scheme memory heap).

It is possible to lock a section of the managed heap by using the C# fixed statement. This is used so that a section of the managed heap can be passed to unmanaged code without worrying that a future GC will move the memory that the unmanaged code is operating on. However, this is completely under the control of the programmer, and is not how Platform Invoke works.

During a P/Invoke call the runtime doesn't mimic the C# fixed statement. Instead, classes and structures (everything of consequence) are marshaled to native code through the following pseudo-process:

The runtime allocates a chunk of unmanaged memory.
The managed class data is copied into the unmanaged memory.
The unmanaged function is invoked, passing it the unmanaged memory information instead of the managed memory information. This must be done so that if a GC occurs, the unmanaged function doesn't need to worry about it. (And yes, you need to worry about GCs, as the unmanaged function could call back into the runtime, ultimately leading to a GC. Multi-threaded code can also cause a GC while unmanaged code is executing.)
The unmanaged memory is copied back into managed memory.

See Class and Structure Marshaling [^] for more detailed information about marshaling classes and structures.

There is one key point to keep in mind: the memory management specified in the above process is implicit, and there is no way to control how the runtime allocates the marshaled memory, or how long it lasts. This is crucial. If the runtime marshals a string (e.g. UTF-16 to Ansi conversion), the marshaled string will only last as long as the call. The unmanaged code CANNOT keep a reference to this memory, as it WILL be freed after the call ends. Failure to heed this restriction can result in "strange behavior", including memory access violations and process death. This is true for any marshaling process where the runtime allocates memory for the marshal process.

The one pseudo-exception to this point is with delegates. The unmanaged function pointer that represents the managed delegate lasts as long as the managed delegate does. When the delegate is collected by the GC, the unmanaged function pointer will also be collected. This is also important: if the delegate is collected and unmanaged memory invokes the function pointer, you're treading on thin ground. Anything could happen, including a process seg-fault. Consequently, you must ensure that the lifetime of the unmanaged function pointer is a proper subset of the lifetime of the managed delegate instance.

Top

Strings

String [^]s are special. String marshaling behavior is also highly platform dependent.

String marshaling for a function call can be specified in the function declaration with the DllImport attribute, by setting the CharSet [^] field. The default value for this field is CharSet.Ansi [^]. The CharSet.Auto [^] value implies "magic."

Some background. The Microsoft Win32 API supports two forms of strings: "ANSI" strings, the native character set, such as ASCII, ISO-8859-1, or a Double Byte Character Set such as JIS; and Unicode strings, originally UCS-2, and now UTF-16. Windows supports these string formats by appending an "A" for Ansi string APIs and a "W" ("wide") for Unicode string APIs.

Consider this Win32 API description:

   [DllImport ("gdi32.dll", 
      CharSet=CharSet.Auto, 
      CallingConvention=CallingConvention.StdCall)]
   private static extern bool TextOut (
      System.IntPtr hdc,
      int nXStart,
      int nYStart,
      string lpString,
      int cbString);

When TextOut is called, the "magic" properties of String marshaling become apparent. Due to string marshaling, the runtime doesn't just look for an unmanaged function with the same name as the specified method, as specified in Invoking Unmanaged Code [^]. Other permutations of the function may be searched for, depending on the CLI runtime and the host platform.

There are three functions that may be searched for:

TextOutW for Unicode string marshaling
TextOutA for Ansi string marshaling
TextOut with the platform-default marshaling

Under .NET, for platforms whose default character set is UCS2 or UTF-16 Unicode (all flavors of Windows NT, and Windows XP), the default search path is TextOutW, TextOutA, and TextOut. Unicode marshaling is preferred, as (ideally) the System.String can be passed as-is to the function, as long as the function doesn't modify the string parameter. Windows CE does not look for TextOutA, as it has no Ansi APIs.

Under .NET, for platforms whose default character set is Ansi (Windows 9x, Windows ME), the default search path is TextOutA and TextOut (TextOutW is not looked for). Ansi marshaling will require translating the Unicode string into an 8-bit string (or DBCS, depending on the country) in the user's locale. Most (all?) of the time, this WILL NOT be UTF-8, so you CAN NOT assume that CharSet.Ansi will generate UTF-8-encoded strings.

TODO: what does Mono do on Unix?

If you don't want the runtime to search for the alternate unmanaged functions, specify a CharSet value other than CharSet.Auto. This will cause the runtime to look only for the specified function. Note that if you pass a wrongly encoded string (e.g. calling MessageBoxW when the CharSet is CharSet.Ansi, the default), you are crossing into "undefined" territory. The unmanaged function will receive data encoded in ways it wasn't expecting, so you may get such bizarre things as Asian text when displaying "Hello, World".

Perhaps in the future the CharSet [^] enumeration will contain more choices, such as UnicodeLE (little-endian), UnicodeBE (big-endian), Utf7, Utf8, and other common choices. Additionally, making such a change would also likely require changing the UnmanagedType enumeration. However, these would need to go through ECMA, so it won't happen next week. (Unless some time has passed since this was originally written, in which case it may very well be next week. But don't count on it.)

Top

More Control

Using the DllImport attribute works if you want to control all the strings in a function, but what if you need more control? You would need more control if a string is a member of a structure, or if the function uses multiple different types of strings as parameters. In these circumstances, the MarshalAs attribute can be used, setting the Value [^] property (which is set in the constructor) to a value from the UnmanagedType [^] enumeration. For example:

   [DllImport ("does-not-exist")]
   private static extern void Foo (
      [MarshalAs(UnmanagedType.LPStr)] string ansiString,
      [MarshalAs(UnmanagedType.LPWStr)] string unicodeString,
      [MarshalAs(UnmanagedType.LPTStr)] string platformString);

As you can guess by reading the example, UnmanagedType.LPStr will marshal the input string into an Ansi string, UnmanagedType.LPWStr will marshal the input string into a Unicode string (effectively doing nothing), and UnmanagedType.LPTStr will convert the string to the platform's default string encoding. For all flavors of Windows NT (Windows NT 3.51 and 4.0, Windows 2000, Windows XP, Windows Server 2003) the platform default encoding is Unicode, while for all Windows 9x flavors (Windows 95, 98, ME) the platform default encoding is Ansi.

There are other UnmangedType string marshaling options, but they're primarily of interest in COM Interop (BStr, AnsiBStr, TBStr).

TODO: Does anybody know the default encoding for Unix platforms? I would guess Ansi as well...

If UnmanagedType doesn't provide enough flexibility for your string marshaling needs (for example, you're wrapping GTK+ and you need to marshal strings in UTF-8 format), look at the Custom Marshaling [^] section.

Passing Caller-Modifiable Strings

A common C idiom is for the caller to provide the callee a buffer to fill. For example, consider strncpy(3):

   char* strncpy (char *dest, const char *src, size_t n);

We can't use System.String for both parameters, as strings are immutable. This is OK for "src", but "dest" will be modified, and the caller should be able to see the modification.

The solution is to use a System.Text.StringBuilder [^], which gets special marshaling support from the runtime. This would allow strncpy(3) to be wrapped and used as:

   [DllImport ("libc.so")]
   private static extern void strncpy (StringBuilder dest, 
      string src, uint n);
   private static void UseStrncpy ()
   {
      StringBuilder sb = new StringBuilder (256);
      strncpy (sb, "this is the source string", sb.Capacity);
      Console.WriteLine (sb.ToString());
   }

Some things to note is that the return value of strncpy(3) was changed to "void", as there is no way to specify that the return value will be the same pointer address as the input "dest" string buffer, and thus it doesn't need to be marshaled. If "string" were used instead, Bad Things could happen (the returned string would be freed; see Strings as Return Values [^]). The StringBuilder is allocated with the correct amount of storage as a constructor parameter, and this amount of storage is passed to strncpy(3) to prevent buffer overflow.

TODO: how does StringBuilder interact with the specified CharSet?

Top

Strings as Return Values

The String type is a class, so see the section on returning classes from functions [^].

If you don't want the runtime to free the returned string, either (a) don't specify the return value (as was done for the strncpy(3) function above), or (b) return an IntPtr [^] and use one of the Marshal.PtrToString* functions, depending on the type of string returned. For example, use Marshal.PtrToStringAnsi [^] to marshal from a Ansi string, and use Marshal.PtrToStringUni [^] to marshal from a Unicode string.

Top

Class and Structure Marshaling

The conceptual steps that occur to marshal classes and structures is detailed above, in the Memory Boundary [^] section.

The principal difference between class and structure marshaling is which, if any, of conceptual steps actually occurs. :-)

Top

Class Marshaling

Remember that classes are heap-allocated and garbage-collected in the CLI. As such, you cannot pass classes by value to unmanaged functions, only by reference:

   /* Unmanaged code declarations */
   struct UnmanagedStruct {
      int a, b, c;
   };
   void WRONG (struct UnamangedStruct pass_by_value);
   void RIGHT (struct UnmanagedStruct *pass_by_reference);

This means that you cannot use classes to invoke unmanaged functions that expect pass-by-value variables (such as the WRONG function, above).

There are two other issues with classes. First of all, classes by default use Auto [^] layout. This means that the ordering of class data members is unknown, and won't be determined until runtime. The runtime can rearrange the order of members in any way it chooses, to optimize for access time or data layout space. As such, you MUST use the StructLayoutAttribute [^] and specify a LayoutKind [^] value of Sequential [^] or Explicit [^].

Secondly, classes (again, by default) only have in-bound marshaling. That is, Step 4 (copying the unmanaged memory representation back into managed memory) is ommitted. If you need the unmanaged memory to be copied back into managed memory, you must addorn the DllImport function declaration argument with an Out [^] attribute. You will also need to use the In [^] attribute if you want copy-in and copy-out behavior. To summarize:

Using [In] is equivalent to not specifying any parameter attributes, and will skip Step 4 (copying unmanaged memory into managed memory).
Using [Out] will skip Step 2 (copying managed memory into unmanaged memory).
Use [In, Out] to both copy managed memory to unmanaged memory before the unmanaged function call, and then copy unmanaged memory back to managed memory after the function call.

In some circumstances, the marshaled copy can be omitted. The object will simply be pinned in memory and a pointer to the start of the data passed to the unmanaged function.

TODO: When can this actually occur? If this happened for any class with Sequential layout, you wouldn't need to specify the [Out] attribute, as the unmanaged code would see the actual object. Is there a specific set of circumstances for when this can occur? This appears to happen with StringBuilder (my tests don't require an [Out] to see changes made to the StringBuilder by unmanaged code), but this is the only example I can think of.

Contents

Introduction

Library Handling

Windows DLL Handling

See Also

Linux Shared Library Handling

See Also

Mac OS X Framework and .dylib Handling

See Also

Library Names

Invoking Unmanaged Code

Exception Propagation

See Also

Marshaling

Memory Boundaries

Strings

More Control

Passing Caller-Modifiable Strings

Strings as Return Values

See Also

Class and Structure Marshaling

Class Marshaling