Hi

We have a Transaction Queue file which all users can add records to and one program is dedicated to emptying.

Normally we would build this file to 10 records and the programs which add to it are written to suspend if the file becomes full and continue when there is room. The limit is set at 10 recs so that if something goes wrong with the program that empties we dont have large numbers of records to deal with when the problem is fixed.

Anyway, thats the background.

My problem is that the index of this file keeps indicating full when it isn't.
XCALL ISMROK returns this
IDA'ALC = 0
IDA'AVL = 460
RECSIZ = 4096
KEYSIZ = 6
KEYPOS = 7
DEVDRV = 0
IDX'ALC = -17
IDX'AVL = 23

Obviously its the -17 that causes the problem.
Also the IDA'AVL is now 460 when originally it was 10, so this file has been expanded several times by the auto expand facility.

Is it possible to turn off the auto expand on a file basis?
If not any ideas why the IDX'ALC is -ive?

Indeed, you can disable the auto-expand by adding OPTIONS=NOAUTOX to the miame.ini file.

Whether it or something else is responsible for the garbage in the IDX header is an unknown at this point.

As a possibly related aside, whenever ISAM files are auto-expanded, a note is logged to the ashlog.log file (search for "expand") which might give some clue as to what went wrong here.

In any case, you need to dump and reload that file so we can start with a clean IDX.

Jack, sorry for the interference here, but I think the question was, not completely disable the auto expand removing the option from the miame, but disable it for a specified file; at least was what I understood from "...on a file base"

Thanks Jorge, you're right I was asking if it was possible to disable auto expand for this one file.

I like the auto expand feature it saves our support guys a lot of problems rebuilding full files.

And to answer your comment Jack, we have been deleting and building this file almost on a daily basis.

These are the ashlog.log entries:

24-Aug-07 10:14:18 [p29806-j21] Auto-expand ISAM IDA: DSK0:TQUEUE[100,0] by 50 recs (to 60)
24-Aug-07 10:14:19 [p29806-j21] Auto-expand ISAM IDA: DSK0:TQUEUE[100,0] by 50 recs (to 110)
24-Aug-07 10:14:20 [p29806-j21] Auto-expand ISAM IDA: DSK0:TQUEUE[100,0] by 50 recs (to 160)
24-Aug-07 10:14:21 [p29806-j21] Auto-expand ISAM IDX: DSK0:TQUEUE[100,0] by 10 blocks
24-Aug-07 10:14:21 [p29806-j21] Auto-expand ISAM IDA: DSK0:TQUEUE[100,0] by 50 recs (to 210)
24-Aug-07 10:14:22 [p29806-j21] Auto-expand ISAM IDA: DSK0:TQUEUE[100,0] by 50 recs (to 260)
24-Aug-07 10:14:23 [p29806-j21] Auto-expand ISAM IDA: DSK0:TQUEUE[100,0] by 50 recs (to 310)
24-Aug-07 10:14:24 [p29806-j21] Auto-expand ISAM IDA: DSK0:TQUEUE[100,0] by 50 recs (to 360)
24-Aug-07 10:14:25 [p29806-j21] Auto-expand ISAM IDA: DSK0:TQUEUE[100,0] by 50 recs (to 410)
24-Aug-07 10:14:25 [p29806-j21] Auto-expand ISAM IDX: DSK0:TQUEUE[100,0] by 10 blocks
24-Aug-07 10:14:25 [p29806-j21] Auto-expand ISAM IDA: DSK0:TQUEUE[100,0] by 50 recs (to 460)

Sorry, I missed the "file basis" (probably due to the state of shock induced by the sight of my post-holiday-weekend inbox).

Unfortunately, while I imagined the possibility of someone not wanting the auto-expand capability, I didn't envision the possibility that it might be wanted on a file-by-file basis. (This would require some kind of table to be checked whenever the need arose.)

There is a workaround though: You can turn the OPTIONS flags on/off at runtime, using the MIAMEX functions MX_GETOPTIONS(59) and MX_SETOPTIONS(60).

So if access to this particular file was localized in one module, it might be practical to retrieve the options and then clear the GOP_AUTOX flag just prior to accessing the file, then restore it just after the access.

If that's not practical, let me know and I'll consider the alternatives.

As for what is causing the IDX to get corrupted, I guess it remains a mystery. Your log seems to suggest fairly heavy use of the auto-expand, so if there were a problem with it resulting in IDX corruption in general, I think you (or any of the others using this option) would be seeing it.

If the above workaround doesn't work (is unworkable or doesn't solve the corruption issue), then the next step would be to turn on TRACE=ISAM (again, you can do this under program control to limit the amount of tracing, using MX_GETTRACE and MX_SETTRACE) which outputs some details which might help establish just what is happening with the file and when it is getting corrupted.

Unfortunatly the workarround is unworkable, there are numerous programs that post to this file and quite a few that could empty it.

A couple of points (alright 3):
This file used to be a 512 byte record, did get auto expanded, but never caused a problem.
This is the only 4096 byte file we have.
The file has been rebuilt to 500 recs and has not caused a problem since set at this size (also has not been auto expanded).

In the ashlog file is a program name, as we do have a program of this name which could add a number of recs rapidly? If this is the case then we could turn trace on for this program when next run.

Will the -ive number in the IDX'ALC cause a problem during file use or could it just be ISMROK returning an incorrect number? The file hasn't caused any program errors or aborts, as far as I know, just the -ive number causes other programs to think the file is full/corrupt and halt.

Ahh, adding features would be so much easier if one didn't have to:

a. Search one's soul to determine whether it makes more sense to default the feature on or off.

b. Provide a way to turn it on and off at the site level.

c. Provide a way to turn it on and off from within a program.

d. Provide ways to turn it on and off automatically based on environmental factors, such as which ppn, user, or which file was being operated on.

e. Provide tracing so that one can tell if the feature is really working, and possibly debug it if it isn't.

f. Make sure that any enhancements do not create backward compatibility issues.

g. Document all of the above, in the various manuals which apply - setup guide, reference, release notes, etc., and index it so that it might someday be found.

h. Wait 5 years for someone to update to the new feature and then find issues with it, requiring mental time travel to recover the necessary knowledge and insights which were once fresh.

(Not that any of the above should be interpreted as being directed at anyone in particular; I'm sure we all face the same difficulties. I just thought I would let out a little steam, badly needed after a week of 110F+ weather here.)

OK, back to business:

1. I've identified that there is a bug in ISMROK.SBR and/or in the autoexpansion which is responsible for the negative IDX allocation value. The bug does not appear to affect the actual integrity of the rock, as ISMDMP still reports the IDX freelist pointer and free count correctly. (ISMROK is performing some arithmetic on those two values to come up with the "number of allocated IDX blocks"), but that value appears to hold no sway in the internal workings of ISAM. So we can expect a bug fix for that shortly. (Independent of that, you might just decide that you can ignore the "Number of allocated IDX blocks" figure returned by ISMROK.SBR. It's not clear just what you would do with the information, other than perhaps checking for anomalies.)

2. The name in angle brackets in the ashlog.log, e.g. is the program name that is currently running. Feel free to run your test (you can turn the trace on with SET TRACE ISAM ON), but since I can reproduce the problem now, there is probably no need for you to spend further time on debugging it.

3. I'll think about the problem of how to enable auto-expansion on a file-by-file basis, without creating too much complication for the relatively subtle benefit. If, in the meantime, you decide that it would be just as easy to enforce your file size limit by some other means, please say so.

I'll be on a business trip for a week starting tomorrow, so there is no chance of any such feature getting implemented until after that. I could, however, nail the bug before then, if it would be useful to you.

Here's a possible solution (although I have no idea how complex it would be to implement). Add support for an alternate ISAM file extension (e.g., ISX/ISA). Files with this other extension would not be subject to Auto Expansion, but would be treated identically to IDX/IDA in all other respects.

Thanks for the suggestion - it would most likely simpler than maintaining an arbitrary-sized list somewhere. I'll consider it.

While fixing the problem with ISMROK.SBR (which was really more of a bug in the ISAM auto-expand, even though the effect was only felt by ISMROK.SBR), I decided on an even simpler scheme for disabling auto-expand on an individual file basis:

If there exists a file by the name .NOX in the same directory as the .IDA or .IDX, then the auto-expand for that file will be disabled. (If the IDA and IDX files are in different directories, or if there are multiple IDX files, you may need multiple .NOX files; or, alternately, you could disable the auto-expand for just one IDX or for just the IDA if you really wanted to get sophisticated.)

It's a bit of a hack, but relatively clean and doesn't require much code to support it.

This behavior is included in 5.0.995.4, which is available as a .tz file in the various Linux and Windows directories beneath http://www.microsabio.net/dist/50rel/.

I did not post it to the Update Center, as I'm somewhat concerned that anything affecting file i/o ought to undergo more serious testing before unleashing it on unsuspecting users who might otherwise get it via ASHUPD without warning. So in-house testing is advised. (I tried it on my own torture test, but a little more field testing would be advisable before distribution to end-user sites.

Please see the http://www.microsabio.net/dist/50rel/ashdevnotes.txt for details.

Actually all I wanted was to get the bug fixed...
I was prepared to live with the auto-expand, even on the file where I didn't really want it, as it's such a wonderful, useful feature.

But hey a bonus - being able to switch off auto-expand on a particular file is much, much more than I expected.

We have downloaded the update, will try it, and let you know in a few days.

ps. Does 110F make you cranky? It never gets that hot here.
By the way a comfortable 23C (73F) here today.
Oh and the aircon is on...

got AirCon? Cool, can I drive 30 miles south'ish and come work in your office today? (We have to use manual Aircon here - Windows wide open)

Jack, I commend you on your simple and clever solution! You can further avoid conflict if the non-expansion extension is defined as a MIAME.INI parm, with *.NOX as the default if no parm is present.

Thank you Steven - it was actually your suggestion that led me to it. As for a configurable option to set the special extension, I guess if anyone reports that they use .NOX (perhaps some kind of smog-tracking file?) for other purposes, it remains an option.

As for aircon, we have it too (L.A. would be deserted without it), but when it gets to 110F, the unit just displays a status code on its LED, which, according to the manual, means "You've got to be kidding!" (That's when I display my own status code indicating "I'm now cranky.")

On the other hand, yesterday a "cold" front arrived and it barely reached 90, so life is good right now (or as good as it can be on the verge of a trip to Omaha...)

Y'all enjoy your warm and pleasant weather - when the Greenland ice sheet melts, they say it's going to get awfully cold in England. (I may even want to move there.)

We used to use that brand of aircon but these damn computers kick out too much heat!