A little background is needed here...
A-Shell/Windows function key handling is similar to AMOS's, except that the PCTDVG driver is built-in, and it skips the first stage of the translation.
With a real terminal or emulator under AMOS, hitting a function key typically sends a multi-byte sequence starting with ESC, chr(155),or sometimes chr(1). For example, F11 might send chr(155) followed by chr(134). If the application wants to do function key translations, it sets a bit in the terminal status word which causes the terminal driver to trap these multi-byte sequences and convert them into single byte sequences with the 8th bit set. Usually that is just a matter of discarding the leadin character and if necessary, setting the 8th bit on the next character, although the details may vary among terminals and drivers. So in this scenario, what the application would see when the user hits, say F11, might be chr(134).
That's the first stage of the translation process, but it isn't sufficient to achieve terminal independence because there never was a standard for what a particular function key should send, either in the raw multi-byte sequence or in the partially cooked single-byte output.
The second step is to pass the partially cooked byte through a translation table to get a fully-cooked translation (which could be any number of characters). Under AMOS, this was done with translation tables generated by either FIXTRN.LIT or SET PFK. In both types of translation tables, the file name is supposed to match the terminal driver. SET PFK always uses .PFK for the extension, while FIXTRN extensions are meant to be application specific (i.e. .VUX for VUE, .IFX for INFLD, etc.)
A-Shell/Windows fully emulates that second step of the translation process, although it uses FIXTRN.LIT for both types. Let's focus on the INFLD .IFX files. As under AMOS, these will be stored in the LIB: directory, aka DSK0:[7,0]. For A-Shell/Windows and the PCTDVG driver, the INFLD (IFX) table is called PCTDVG.IFX. It is also the default, so we can edit it by just using FIXTRN with no arguments:
.FIXTRN
Editing function key translation file: LIB:PCTDVG.IFX
Enter delimiter key (cannot be used in a translation): \
(It echoes the LIB:PCTDVG.IFX file name to confirm which file is being edited. I typically use \ as the delimiter key. Now you just hit a function key and it displays the current translation, if any, and gives you the opportunity to change it
Hit a function key (or special delimiter to exit): [181] <-- I hit F1 here
Current Translation: ^G^A
Update? [y/N]
Hit a function key (or special delimiter to exit): [133] <-- I hit F10 here
Current Translation: ^G^J
Update? [y/N]
Hit a function key (or special delimiter to exit): [134] <-- I hit F1 here
Current Translation: ^G^K
Update? [y/N]
As you can see from the above, F1 transmits the partially cooked value chr(181), which the translation table then converts to ^G^A. For F11, the partially cooked value is chr(134) and the full translation is ^G^K.
This is the standard INFLD translation scheme, since INFLD interprets Control-G followed by a byte chr(X) by setting exitcode -X. So ^G^A gets turned into exitcode -1, ^G^J into -10, and ^G^K into -11.
There's no law saying you have to use these sequences, unless you expect that function key F# will generate exitcode -#, which is typically the expectation with INFLD.
With that behind us, we should be able to get the bottom of the problem with F11 under A-Shell/Windows (not ATE) by just using the above procedure. If it turns out that F11 is being translated into something else, then you can just correct it by saying "Y" to the "Update? [y/N]" question, and typing Control-G Control-K (followed by the delimiter key \). Then hit the delimiter key again to exit the program, and it will automatically save the updated LIB:PCTDVG.IFX and offer to update the copy in memory.
(As an aside, it's a bit mysterious why your PCTDVG.IFX file would not match the standard one in the release, unless perhaps it got modified, or there is some problem with the version in the release, in which case we should get that resolved.)
For ATE, it's a bit more complicated, because it is attempting to act more like a standard terminal/emulator, meaning that it will send the raw multi-byte sequence to the server and let the server handle converting it back to a partially cooked single byte, and then into a fully cooked sequence via the appropriate translation table. But, since ATE is actually an A-Shell/Windows application, it starts with the partially-cooked single byte translations, and then uses the local (client-side) translation table system to convert it to the appropriate multi-byte sequence to transmit to the server, for the emulation you've selected.
Normally you don't need to think about that, because ATE comes with a set of .PFK files appropriate for the various emulations that you can select in the ATE connection configuration dialog. For example, if you set your ATE emulation to AM75G, then it will use the local AM75G.PFK file to translate the partially-cooked function key values issued by A-Shell/Windows, into the multi-byte sequences expected by the AM75 terminal driver on the server.
If the end result isn't as expected, I would start with the above procedure on the server side to see if you can resolve it there. But if you find that hitting, say F11, results in a message from FIXTRN that it isn't a function key (i.e. it doesn't result in a partially cooked single byte between 129 and 255), then you may need to look at the .PFK file on the local PC. That's generally not something that I recommend fooling around with, since it's hard for you to know what the emulation is really supposed to send, and it needs to match what the driver on the server side is expected. And because there are multiple steps involved, some on the client side and some on the server side, it's easy to get confused. Plus, for a given emulation, the client-side translation should be standard (i.e. should match the standard for that emulation). So it's best to just contact me directly so I can help you get it sorted out.
Note that ATE supports the following emulations:
AM75, AM75G, AM65, AM65G, WYSE50, WYSE5G, AM62CG. Those are also the emulations supported by ATSD. A-Shell/UNIX supports a few others in addition, including VT320. Like most standards, there are always variations, so it is possible that a particular emulation X in a particular emulator Y doesn't match the emulation behavior in some other emulator. Naturally, the ATE emulations should match up well with the A-Shell built-it drivers on the server side. They might not match up quite as well with some other emulator, although I'm pretty sure there are a lot of copies of ZTERM and AlphaLAN out there connecting to various flavors of A-Shell, as well as copies of ATE connecting to non-A-Shell servers (including AMOS). In any case, if there's a mismatch, we'll try to figure out a resolution, but we'll need "all the details".