Java calling hook via jni is invalid

I need to implement a hook program to disable mouse and keyboard input, then generate it as dll, and then call it in java. When calling, the program does not report an error, but the call is invalid. Here is part of the code

cPP section:

// dllmain.cpp : Defines the entry point for the DLL application.
-sharpinclude "stdafx.h"
-sharpinclude "stdio.h"

HHOOK g_HookHwnd = NULL;
HHOOK g_hMouse = NULL;

// 
extern "C" _declspec(dllexport) LRESULT CALLBACK MyHookFun(int nCode, WPARAM wParam, LPARAM lParam)
{
    printf("in hook key function\n");
    // Structure
    /*typedef struct {
    DWORD vkCode;
    DWORD scanCode;
    DWORD flags;
    DWORD time;
    ULONG_PTR dwExtraInfo;
    } KBDLLHOOKSTRUCT, *PKBDLLHOOKSTRUCT;*/
    // vkCode
    PKBDLLHOOKSTRUCT pVirKey = (PKBDLLHOOKSTRUCT)lParam;

    // MSDN,nCode < 0
    if (nCode >= 0)
    {
        // 
        switch (wParam)
        {
        case WM_KEYDOWN:
        case WM_SYSKEYDOWN:
        case WM_KEYUP:
        case WM_SYSKEYUP:
            switch (pVirKey->vkCode)
            {
            case VK_LWIN:
            case VK_RWIN:
                return 1;  // 
                break;
            }
            return 1;
            break;
        }
    }

    return CallNextHookEx(g_HookHwnd, nCode, wParam, lParam);
}

extern "C" _declspec(dllexport) LRESULT CALLBACK  MyHookMouse(int nCode, WPARAM wParam, LPARAM lParam)
{
    printf("in hook mouse function\n");
    return 1;
}


HMODULE g_Module;
extern "C" _declspec(dllexport)  BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
    {

        break;
    }
    }
    return TRUE;
}


extern "C" _declspec(dllexport) void FuncHookDevice()
{
    if (!g_HookHwnd)
    {
        printf("start hook\n");
        g_HookHwnd = SetWindowsHookEx(WH_KEYBOARD_LL, MyHookFun, g_Module, 0);
        //g_hMouse = SetWindowsHookEx(WH_MOUSE_LL, MyHookMouse, g_Module, 0);  //
    }
}

extern "C" _declspec(dllexport) void FuncEndHook()
{
    printf("end hook\n");
    UnhookWindowsHookEx(g_HookHwnd);
    //UnhookWindowsHookEx(g_hMouse);   //
    g_HookHwnd = NULL;
}

java Program:

public class HookTest {
    public interface  Hook extends Library{
        Hook INSTANCE = (Hook) Native.loadLibrary("lib/Hook",Hook.class);
        public void FuncEndHook();
        public void FuncHookDevice();
    }

    public static void main(String[] args) {
        Hook.INSTANCE.FuncHookDevice();
    }
}

run result:
clipboard.png

After running the

program, I found that only "start hook" was output and then the program stopped running. The "in hook key function" is not printed out. It shows that the program does not enter the inside of the hook function. I have tested the hook of this cPP program. Using cPP to call dll can successfully disable the keyboard or mouse.

so why is it invalid when called with java.

Mar.04,2021

finally solved this problem, and found an answer to the question on stackoverflow: Q & A details

the practice is to add a message queue. Why is it added to this question and answer: Why do you add a message queue , roughly because the low-level mouse and keyboard hook will return the return value immediately by default, which can be solved by adding a message queue.

added a thread to control the hook. If you don't add it, it seems that the program cannot exit as required.

Last part of cPP code:

// dllmain.cpp : Defines the entry point for the DLL application.
-sharpinclude "stdafx.h"
-sharpinclude "stdio.h"

HHOOK g_HookHwnd = NULL;
HHOOK g_hMouse = NULL;
HANDLE hThread = NULL;

// 
extern "C" _declspec(dllexport) LRESULT CALLBACK MyHookFun(int nCode, WPARAM wParam, LPARAM lParam)
{
    printf("in hook key function\n");
    // Structure
    /*typedef struct {
    DWORD vkCode;
    DWORD scanCode;
    DWORD flags;
    DWORD time;
    ULONG_PTR dwExtraInfo;
    } KBDLLHOOKSTRUCT, *PKBDLLHOOKSTRUCT;*/
    // vkCode
    PKBDLLHOOKSTRUCT pVirKey = (PKBDLLHOOKSTRUCT)lParam;

    // MSDN,nCode < 0
    if (nCode >= 0)
    {
        // 
        switch (wParam)
        {
        case WM_KEYDOWN:
        case WM_SYSKEYDOWN:
        case WM_KEYUP:
        case WM_SYSKEYUP:
            switch (pVirKey->vkCode)
            {
            case VK_LWIN:
            case VK_RWIN:
                return 1;  // 
                break;
            }
            return 1;
            break;
        }
    }

    return CallNextHookEx(g_HookHwnd, nCode, wParam, lParam);
}

extern "C" _declspec(dllexport) LRESULT CALLBACK  MyHookMouse(int nCode, WPARAM wParam, LPARAM lParam)
{
    printf("in hook mouse function\n");
    return 1;
}


HMODULE g_Module;
extern "C" _declspec(dllexport)  BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
    {

        break;
    }
    }
    return TRUE;
}


extern "C" _declspec(dllexport) DWORD WINAPI controllHook(LPVOID lpParamter)
{
    g_HookHwnd = SetWindowsHookEx(WH_KEYBOARD_LL, MyHookFun, g_Module, 0);
    g_hMouse = SetWindowsHookEx(WH_MOUSE_LL, MyHookMouse, g_Module, 0);
    while (true) {
        MSG msg;
        if (PeekMessage(&msg, 0, 0, 0, PM_REMOVE)) {
            TranslateMessage(&msg);
            DispatchMessage(&msg);
        }
    }
}


extern "C" _declspec(dllexport) void FuncHookDevice()
{
    if (!g_HookHwnd)
    {
        hThread = CreateThread(NULL, 0, controllHook, NULL, 0, NULL);
        
        printf("start hook\n");
        g_HookHwnd = SetWindowsHookEx(WH_KEYBOARD_LL, MyHookFun, g_Module, 0);

    }
}

extern "C" _declspec(dllexport) void FuncEndHook()
{
    printf("end hook\n");
    UnhookWindowsHookEx(g_HookHwnd);
    UnhookWindowsHookEx(g_hMouse);
    CloseHandle(hThread);
    g_HookHwnd = NULL;
}
Menu