I have a problem with calling function in custom dll written on C (or C++).
The prototype of function: extern unsigned short _stdcall InitRndm (char *key_dev, int window_enable );
DLL name: testrndm.dll
This is tricky, since I can't test with your code. However, I'm not sure where your InitRndmDelegate is coming from; I don't see anything about a delegate in the C definition. Is your C function expecting an Ascii-encoded string? I also don't tend to use -PassThru with Add-Type, so I'll stick to what I'm familiar with in this reply.
I would try this:
Add-Type -TypeDefinition @'
public static class TestRndm
[DllImport("testrndm.dll", EntryPoint = "InitRndm", CharSet = CharSet.Ansi)]
public static extern short InitRndm(string key_dev, int window_enable);