Which plugin are you using to display the dialog, the enhancement pack one?
The only other way I have seen the string inputed is with null characters instead of pipes ("|") and two null characters at the end of the string.
Try
filter = "DirectX Files"+chr$(0)+"*.X"+chr$(0)+chr$(0)
From
msdn
Quote: "lpstrFilter
Pointer to a buffer containing pairs of null-terminated filter strings. The last string in the buffer must be terminated by two NULL characters.
The first string in each pair is a display string that describes the filter (for example, "Text Files"), and the second string specifies the filter pattern (for example, "*.TXT"). To specify multiple filter patterns for a single display string, use a semicolon to separate the patterns (for example, "*.TXT;*.DOC;*.BAK"). A pattern string can be a combination of valid file name characters and the asterisk (*) wildcard character. Do not include spaces in the pattern string.
The system does not change the order of the filters. It displays them in the File Types combo box in the order specified in lpstrFilter.
If lpstrFilter is NULL, the dialog box does not display any filters.
Windows XP: In the case of a shortcut, if no filter is set, GetOpenFileName and GetSaveFileName retrieve the name of the .lnk file, not its target. This behavior is the same as setting the OFN_NODEREFERENCELINKS flag in the Flags member. To retrieve a shortcut's target without filtering, use the string "All Files\0*.*\0\0"."

it's cool to hate