How TaskDialogIndirect() can turn Cancel into Yes

User Interface designers are rather sadistic. In the year 3000, when suicide booths will be commonplace, a User Interface designer will be the sort of fellow who places the “Stab Me to Death” button adjacent to the “Enjoy a Lovely Cup of Tea” button on the Slurp ‘N Spear vending machine down at the local Automat. If you’re interested in getting repeat business, you’ll probably want to confirm that the consumer would rather purchase a gut full of twisting metal instead of a mouthful of dried leaves in water.

Consider the following code analog to this situation, where the Delete button looks too much like, or is placed to close to the Save button in a toolbar. As a defensive programmer you safeguard the delete function with a confirmation:

  if MessageDlg('This action is not undoable.'#13 +
    'Are you sure you want to delete all your work?',
    mtConfirmation, [mbYes,mbNo], 0) = mrNo then
    exit;

  DoDeleteAllYourWork();

This works fine and the early bailout situation is preferable in my book when the code of DoDeleteAllYourWork() actually follows inline. The code is written, compiled, shipped and you forget all about it until a few years later someone calls up spitting venom because they pressed Escape and your app deleted their work. What has occurred is a subtle issue created by Delphi switching from their own MessageDlg to Vista’s powerful TaskDialogIndirect() API when you enable runtime themes in your project options.

The little check with 100 implications

So what does Enable Runtime Themes actually do?
Checking this box actually tells Delphi to compile a manifest into your application. A manifest is a text XML resource of type RT_MANIFEST (24), which specifies the activation context of your application. In the case of the enable runtime themes and Delphi, this creates a CREATEPROCESS_MANIFEST_RESOURCE_ID (1) type manifest to request a version 6 COMCTL32.dll– the version that supports XP-style theming. This has a few implications (some dependent on windows version)

• All your controls coordinate better (“theme”) with the desktop.
• Teaches you what the ParentBackground property means.
• Directory and registry virtualization is disabled, meaning your shitty code writing to HKEY_LOCAL_MACHINE, %ProgramFiles%, etc will stop silently writing to %APPDATA%\Local\VirtualStore and start failing.
• Alters the behavior of of MessageDlg.

How does MessageDlg differ with theming enabled on Vista and above?
Delphi used to build the dialog created by MessageDlg itself, actually constructing a TForm descendant called TMessageForm and programmatically creating the buttons and labels on it. Microsoft however included a powerful API starting with Vista that allows you to create simple themed Yes/No or selection dialogs with very little code. Once you turn on theming, Delphi will start using these style dialogs on Vista (Windows Version 6) and above instead of creating its own TMessageForm classes.

  // Excerpt from MessageDlg in Delphi 2007
  if (Win32MajorVersion >= 6) and UseLatestCommonDialogs and ThemeServices.ThemesEnabled then
    Result := DoTaskMessageDlgPosHelp('', Msg, DlgType, Buttons,
      HelpCtx, X, Y, HelpFileName, DefaultButton)

The difference is, and here’s where your tea-desiring patron gets stabbed in the gut, they behave slightly differently with regards to what is returned when the user cancels. The code at the beginning of this article works as expected when running with theming turned off, but starts deleting data when theming is turned on. What’s happening is Windows returns IDCANCEL if “Cancel button was selected, Alt-F4 was pressed, Escape was pressed or the user clicked on the close window button.“. This behavior can occur regardless of if there is a cancel button on the dialog. The simple solution is to just test for the mrYes result or check for both negative options

  if MessageDlg('This action is not undoable.'#13 +
    'Are you sure you want to delete all your work?',
    mtConfirmation, [mbYes,mbNo], 0) in [mrCancel,mrNo] then
    exit;

  DoDeleteAllYourWork();

This is a rather simple problem with an extremely simple solution, but one of many gotchas that comes with requesting theming. If you’re interested in knowing more about TaskDialogIndirect(), check out the Delphi TTaskDialog class, or the simple TaskMessageDlg() which is a MessageDlg with an better looking title line. Welcome to the world of ::MessageBox() 2.0!