As it stands, we can only create one hook per file path.

I'm starting to design a refactor around how we deploy MX_FILEHOOKs, and I'm wanting to build in the ability to have multiple handlers for a given file. As of now we mostly use file hooks for SQL DB data synchronization, but we'd like to be able to add a generic logging handler, and perhaps a debug logging handler. For code quality/maintainability I want these handlers to be separate SBXs.

We could build into our framework the ability to define multiple handlers per file with different hook events and flags. It would create one hook with all the necessary events and flags, and then use one wrapper handler that was responsible for calling each actual handler based on the current matching events/flags.

It seems before we do that, the community might benefit from this being simpler to implement and not having to reinvent the wheel on our own?

I also have some performance apprehension with using a wrapper handler written in BASIC as opposed to the C implementation. In this area microseconds add up.

P.S. It's probably a pipe dream, but file hook handler calls being optionally asynchronous would be a potentially huge performance benefit. It would also make it some what easier to implement our database synchronization. As it stands, we have a handler that just writes a binary file with only the hook event data and current record in it. There's a separate background process that monitors the directory of these binary files, and handles updating the DB from them. This is a large performance gain (opposed to updating the DB from the handler SBX), but it's much more time consuming to implement. Plus you also now have to handle the life cycle of the background process. If we could get away with updating the DB from the handler SBX it would be much simpler.

Both are interesting ideas.

I guess I'd prefer to leave the problems associated with multiple hooks per file (establishing and dealing with order dependencies, overlap, etc.) to the application rather than make A-Shell responsible for it.

Focusing on the second idea - improving the performance (by implementing an asynchronous mechanism or other designs) - would have benefits for all hook users, not just those wanting to implement multi-level handlers.

The existing hook design actually allowed for other types of hook calling mechanisms, but the current SBX scheme was the only one actually implemented. Perhaps it's time to revisit that.

The most straightforward approach would be to implement a "hook service" whereby a background process received hook requests by some asynchronous FIFO means (perhaps TCP or pipe), eliminating the SBX calling overhead and allowing the main A-Shell task to continue running without waiting on the hook response. (You would have to give up the ability for the pre-event hooks to pass back a flag to A-Shell, telling it to skip the actual event.) But some thought would need to go into how to handle the life cycle issues, whether to have the service auto-launched on demand, etc.

Another approach would be to just fork off a child/clone of the current A-Shell task for each hook. That eliminates the complexities of the background task communication and life cycle, but risks creating absurd numbers of processes. And wouldn't work very well in the Windows environment. And wouldn't eliminate the SBX calling overhead. Basically all it would do is change it from the current synchronous to asynchronous.

I'm fine with you not wanting A-Shell to be responsible for multiple hooks. Just wanted to see if there was any other feedback that it would be beneficial to the community at large. Although, I'm not sure how close everyone one follows the forum, so they're not likely to see this until they need to implement it themselves. Oh Well.

It seems like an encrypted TCP connection makes the most sense as that would work for both traditional A-Shell installs and container based installations (docker).

We use filehooks at several customer one who has it hooked up to couple of dozen data files, so any speed increase is always welcome not that we had any feed back that they are slowing the system and programs once enabled.

On each hook our SBX just writes the data file name and record number to x number of sequence files (if pre and post is different), we then have an external widows exe (DVerto) that picks up these sequence files to see what file and record was changed this then reads in the data file over samba and updates a Windows SQL server. In our case this is the slower side and sometimes get a bit of a backlog but that's all unrelated to the Ashell side.

Just to provide a status update on this, I started on implementing a socket-based handler (as an alternative to the current SBX-based handler). The plan is to start out with a handler "service" that receives the socket-based hook requests and forwards them to individual handlers, something like this:



This model would be slower than the existing SBX hooks, except that we'll also introduce an HFF_ASYNC flag that will eliminate the need for the handler to respond, and more importantly for the File I/O system to wait for the hook response. Without a response code, the hook routine won't be able to tell the File I/O system to abort the operation, but that's a pretty obscure feature anyway, and this will allow the application's file i/o to proceed unencumbered by the hook.

Initially I'm going to just launch the socket-based handler manually and do some performance testing to see how viable this is. Assuming all goes well, we might arrange to have A-Shell auto-launch the handler on an as-needed basis.

In the initial model, the socket-based handler is an A-Shell BASIC program that accepts connections from the A-Shell hook subsystem and launches sub-handlers as needed. That provides a lot of flexibility for customizing the behavior of the handler. In fact, you may want to eliminate the sub-handlers and just do all of the handling within the one program. (The trade-off there is between the overhead of launching sub-processes for each I/O operation to be handled, vs. creating a bottleneck in the main TCP link resulting in the application's file i/o being held up. And also between the complexity of the multi-level hierarchy vs. the modularization benefits.)

An alternate design would do away with the socket-based handler and just launch the sub-handlers directly as background tasks. But that works a lot better under *NIX than Windows.

In any case, this is probably going to take a couple of weeks or so to get to something beta-testable, but I just thought I'd let you know the status in case it affected your development plans.

Sounding good and promising

Sounds good. Look forward to seeing details.

I presume this model the "hook" will use a persistent TCP connection. Is that correct?

In order to achieve reasonable performance for a high rate of hook event calls, which is one of the two driving factors in your inquiry, we probably can't tolerate the overhead of an extra connect/disconnect for each event call. So if we're going to use TCP, we probably need persistence.

On the other hand, TCP connections are one-client-to-one-server and thus we need at least as many connections as there are jobs with hooks enabled (and possibly many times more than than if there are different handlers for different hooks/files/events). And that's going to make it difficult to satisfy the other driving factor - design simplicity (modularity, elegance, life-cycle management, etc.)

If we're sure that the event handler will be on the same machine as the application processes are running on, we could consider some TCP alternatives - UDP, named pipes, UNIX domain sockets, etc., but I'm not sure any of those offer enough net advantages over the complications and limitations they are likely to introduce. And besides, having the hook handler on another machine would be very handy in some cases.

But aside from the problem of connection proliferation, we also have the problem of process proliferation. One background process per connection, i.e. per application process? One per hooked file? A pool shared among a greater number of jobs? It's not easy to find a model that scales well across the entire range of possible loads.

Even if the asynchronous case where the application doesn't have to wait for a response from the handler for each event, unless the handler is faster than the application (in many cases it probably won't be), we need a buffering mechanism, else application will be delayed in delivering the events to the handler. (This is an area where named pipes would have a natural advantage, since essentially infinite buffering is built in to the design. TCP sockets, on the other hand, have very limited buffering.)

One possible compromise design that keeps the process proliferation down would be to use a pair of handler processes for each distinct type of hook/file/event -- one to maintain (via multiplexing) persistent connections with all of the jobs currently using that hook/file/event, and the other to do the actual work of the handler. A pipe could be used between the two to provide unlimited buffering, allowing the work to stack up as much as it needs to without necessarily delaying the application processes. (This assumes that a single TCP process multiplexing many connections could keep up with just reading from them and writing to the pipe.)

Suffice to say I'm pondering the matter.

Late to the party, but we've been using MX_FILEHOOKs here for a year or so here so I thought I'd chime in on my usage of them, even if it seems almost identical to how Stephen is already doing it.

I built a generic subroutine to be fired by all hooks. It captures the related data, and saves it to an ISAM file. This program is very light weight, with the intention of having minimal overhead on the program it's interrupting. A separate background process then reads through the data and performs whatever action is required of it, based on some configurable tests and actions. The whole system is fairly modular, and as it's proved pretty flexible so far.

It's great for things that need to be handled sequentially, and is perfect for the initial use it was put to, which was pushing data to a third party system via a web api. Something more asynchronous is going to be a better fit for some of our upcoming projects though, and I've already given some thought to firing of asynchronous sub processes from the main background routine.

Thanks for sharing what sounds like a pretty interesting design. I actually considered using an ISAM file for intermediate buffering, but it would be difficult to allow for totally generic configurations (at the A-Shell level rather than the application level) without some idea of the maximum record sizes needed. It makes much more sense to handle it at the application level, as you've done, since you know what maximum size applies to your situation, you can use whatever scheme you currently use for creating the file, managing it, etc.

And one big advantage of the ISAM scheme is that, unlike pipes or sockets, it's immune to disconnects on either the writing or reading side. So you can shut down your sub-handlers to modify them, without interfering with the capture side. And vice versa. That allows a huge amount of flexibility in sub-handlers without requiring any corresponding accommodations in the main handler, other than indexing the captured records so that different sub-handlers can easily pick out, in chronological order, the ones that are of interest to them.

And the beauty of the SBX-type hook is that it shares the application's process so doesn't require any life cycle management of a secondary processes, like the TCP handler would.

The only further advantages of TCP handler would be:

1) The ability for the hook handler to be on another machine.
2) The ability for the hook handler to be implemented outside of A-Shell.
3) The ability to consolidate the first level of handling into a single process.
4) Lower performance impact on the application.

On the performance point, the main overhead of the SBX interface comes from the interface itself (A-Shell basically has to create a copy of the runtime memory environment for the SBX, and the selectively update the main copy based on changes made to the sub-environment by the SBX). And by having to open and close the ISAM file for each event. (Aside from the impact on each job waiting for the hook handling, for large systems, it means a lot of processes opening and closing that file all the time.)

It might be interesting to try to quantify the overhead, i.e. run some kind of process that generates a lot of hook events, and then time it without and without the hook enabled. But unless the rate of such events is high, it may be a non-issue, since the hook overhead may blend into the general noise/overhead of the application.

It's been a while since I've done any benchmarking, but we have a few outputs that involve writing flags to records as they are read, so you generate a lot of hook events through what the user sees as a query, and I've not seen any significant overheads in those instances.

You're right about the file sizes, the largest ISAM files we'd be interested in monitoring max out at 1024 bytes. To give it full flexibility I've built the intermediate ISAM file to 4096, and store both the contents of the record before the ISAM satatement as well as after it, and some additional meta data that will be useful for auditing purposes. Knowing those limits makes things a lot easier.

The idea of being able to directly fire hook events into TCP or similar is still intriguing though. I think it's the issue of disconnects I'd be most concerned with though. If you need it to be reliable, you're going to need some method of recording failed connections, and managing them. You might find you end up needing something like the ISAM route as a backup just to manage this eventuality.

All of those unanswered questions surrounding the async hooks led to this project being shelved for awhile, but the revival of interest in more efficient ways to export random/isam file updates to an SQL DB (see the thread Random Data File to SQL Table Exporting) has got me thinking again.

Another possibility would be to implement a file-based hook option within A-Shell itself, i.e. one that simply wrote to a transaction log file directly, rather than calling an SBX or sending it to a TCP connection. Compared to the SBX call, it would presumably be faster due to elimination of the SBX call. And compared to the TCP/async scheme, it would be more robust.

For the reasons discussed above it probably isn't practical to use an ISAM file, and for flexibility it probably would have to use variable length records (so the output log would be a binary stream file that would have to be read using something like GET.SBR ).

Aside from speed, another upside would be that you didn't have to implement your own hook routine (although you'd still need to implement a routine to process the transaction log file). (Most of the work there could probably be reduced to a set of common functions though.)

I'm imagining a transaction log which looked something like:

The envelope could contain roughly the same information that was in the Hook SBX spec envelope param. The record data would contain the complete record written for new records, or the old/new data for updated records. It probably makes sense to compress or diff record date to reduce the size of the transaction log.

One downside of such a scheme is that it would probably be impractical for you to update the transaction log file while it was active (like you could with the ISAM scheme described by Tom). But that's probably not a major obstacle. The secondary processing routine would be able to rename the active transaction log in order to process it. In cases where you only wanted to process a subset of the log records at a time, you'd probably just copy the unprocessed records to a secondary log, or update some kind of sidecar status file.

I'd need to do some benchmarks to see how much faster this would be than the present SBX hooks, but if it were, does this seem worth implementing?

As I recall (I think there's a post in regards to this) eliminating the SBX call with the data change flag made a massive difference in performance by not having to call the SBX, so I'd guess there's probably similar performance gains to be made without ever calling an SBX.

What about a mode where each hook event was a separate file named to keep them in sequence?

Sure sounds good to me!

My concern about the one-file-per-event model is that if you have a lot of events and do clean up the files frequently, you're going to end up with a case of directory bloat. (Which, as an aside, is something I continually notice when checking out end-user sites, i.e. directories with tens or even hundreds of thousands of files. This is a performance killer if there ever was one.)

What do you see as the big advantage of a separate file for each event? The ability to easily pick-and-choose which to process? The simplicity of erasing them after processing? The ability to update the files after processing (perhaps to pass information on to subsequent processes for events that are processed in multiple passes)?

That's a good question, I'm not sure I really ever considered using one file. Would we not have to handle rotation to keep if from growing too large? Or, would you remove the record from the file after processing?

I guess was I was imagining is that the ashell->hook->file internal proces would keep adding to the file, similar to the way ashlog works.

Your secondary/backend processor, whenever it ran, would start by renaming the file so as to take ownership of it, which would also force the ashell->hook->file process to create a new file for the subsequent events.

In order to do real-time backend processing, you'd loop, checking to see if the active transaction log file had anything in it; if so, rename it, process the renamed file, and then start over. Otherwise sleep until the active transaction log had data again.

That sounds reasonable

FYI, I have a variation of this new file hook handler close to being ready for beta release. My initial tests suggest that it is maybe twice as fast as a traditional SBX: handler that merely creates a log file. That may not be especially dramatic (it's still significantly slower than with no hook at all), but there are some other advantages which may make it worth further consideration:

- Much lower CPU overhead than the SBX handler. (My tests have been disk-bound so the reduction in CPU overhead didn't really matter much, but in a large multi-user system it might.)

- Simple to activate (no coding required for the log capture operation). You still need to write code for the final back-end processing, but it's easier to debug that code when you are just processing files, compared to when it is running in a real-time synchronous environment.

- Easier to offload the back-end processing to another machine (either via a network mounted directory or file transfers).

It's not yet clear how it fits into the multiple-hooks-per-file request, but since it logs essentially everything that would otherwise be available to an SBX: hook handler, there should be nothing stopping you from processing the log files multiple times with multiple processors.

Anyway, this is just the heads-up; I plan to do more experimentation and perhaps make it available in a day or two.

Sounds mighty good to me, I guess your have some configuration file or a section in miame.ini to say what data files are enabled and updates this log file.

Two features to keep in mind that I had to enabled in my hooks were:
* A global Start/End time so the hooks only processed between 6am and 8pm (for backup reasons/task manager)
* The ability to disable/enable all file hooks (by program) or by a XCall, I had to add the following to start and end of our file reorganises as the hooks were 1) pointless information and 2) slowed things down.

This latest enhancement doesn't change the way hooks are enabled/disabled, which continues to be by XCALL MX_FILEHOOK (or custom wrapper routines like your XCALL MIHOOK.

I agree that the mechanism could probably benefit from ways to make it easier to manage globally, rather than having to go into individual programs to enable/disable specific hooks. Then again, it was never the case that you had to insert your hook activation logic into existing application programs. Once a hook is established for a particular file, it remains live across all programs until it is disabled or deleted. And, MX_FILEHOOK allows you to disable, and later re-enable, all hooks without know which hooks have been set.

So it is certainly possible to create a standalone utility program that takes care of most of your hook management without touching individual programs. EXCEPT for the problem you point out, of certain kinds of housekeeping actions, like reorganizations, which might otherwise generate a tremendous amount of useless hook activity. Those situations will almost certainly require that you explicitly insert code to temporarily disable and re-enable specific hooks, but it's also likely that such housekeeping would only take place in a few dedicate programs.

And for convenience purposes, having an easily editable list of the files to be hooked, perhaps along with some options like automatic activation/deactivation times (apparently like your OPR:MFHOOK.INI), seems like an excellent idea.

At this point, would it be helpful to add such a mechanism, perhaps with a HOOKUTIL.LIT, to the standard package? Or maybe an SOSLIB utility?

I suspect that most of the existing uses for file hooks involve systematic processing of the events, i.e. for replication to external databases, or perhaps for reporting. In those cases the main requirement for the hook event log is that it contain all the information you could possibly want.

But I wonder how many people are interested in just using the file hook mechanism for creating a backup audit trail, sort of the data equivalent to the increasingly ubiquitous security cameras. In this case, you may never even access the hook log (and might just discard them after a set period). But you want to be able to search them, should the need arise, to see when a particular record/field was changed and by whom.

The new LOGFIL: handler is likely to be very attractive for that use, at least for the capture-and-store part of it, since there's almost nothing to do. And, it's more efficient than the previous alternative. The other part, i.e. the searching, might benefit from some standardized tools, although it's not yet clear to me there's an actual need.

Another consideration for this type of hook log would be compactness. The 'squeezed' format offered as an option will typically reduce the log size by about half, but we could probably do much better by focusing just on changes. (That is, rather than log both the prior and new contents of a changed record, we might log just the changed fields.) The main obstacle I see there is that it records physical (bit-level) changes rather than logical (field-level) changes that would probably be of much more interest when searching for who changed the new employee's security clearance.

In theory, the search tool could first reconstruct the entire before/after record contents from the changes (assuming we had a dependable starting point and were sure we had captured ALL changes). But that might be a tall order. So you might end up knowing that a certain user changed a certain bit in the security clearance field, but no being able to tell with 100% certainty what the entire field was prior to, or after the change.

To get around that, the event handler would need to know the record layout for each file being hooked. So maybe that's another enhancement to consider. But like so many of these things, it can get complicated if there are overlays within the file record layout. Or worse, if there are multiple layouts within a file. Maybe we can simplify it though, by treating as "fields" (for the purposes of record field-level changes) only the top level (map2?) variables within the record structure?

I only say as I was thinking sometime down the line I could replace my XCALL MIHOOK with your new pending awesome improved file hook handler, but yes very interested in the Audit part of it too, i dont have fingers to count how many times I would of loved to know what the record data was before user x & y changed it (or rolled back), we have audits for selected files not never enough.

I think we could fairly easy have some cross ref file mapping fields from the random files to the audit data.. The majority of ours are not too complex, (but others maybe..)

Example:
DATAFILE=DSK43:FOBUSR.DAT[241,16]
MAP FILE=FOBUSR.CPY[x,x]
MAP STRUCTURE=FOBUSR

Let me give this a bit more thought. I may go ahead and (beta) release the new LOGFIL: mechanism as is, and then think about refining it to focus on recording field-level changes.

There's nothing stopping you from doing it yourself the old-fashioned way, using either the old SBX: handler or by post-processing the LOGFIL: log files. The main advantages of integrating the record map into it would be:

a) It would reduce the log file size, probably quite substantially, if we only had to record the fields (rather than entire records) that changed. This would be most useful for those just-in-case situations where you just wanted to capture an audit trail on the off-chance that it might someday be needed. (In which case you'd want the files to be as small as possible.)

b) It would reduce the amount of coding needed for each file, possibly to near zero. (We'd have to come up with some standard format for reporting these field-level changes, along with a search program that could handle an entire directory of log files, but that one program might then be sufficient for any inquiry of the type "who changed that field and when?")

The one additional detail that might be needed for such a generic report would be some way to determine how to display a field value. Most of the time it wouldn't be necessary (most string and numeric fields could be displayed as-is); but in some cases you might want to unpack or decode a field's value to make more sense of it. Still even in such a case, the generic search could tell you that at time t, user u changed field f in rec #r from x to y. You could then use your own custom code to make the x and y field values "human-readable".

But just now it occurs to me that knowing the record # might not be sufficient to identify which record this change occurred in. (What if the file gets sorted?) Maybe we also need to identify a record key field that would always be recorded in the log whenever a field changed?

Another detail which occurs to me just now is that to minimize the overhead of the handler (both CPU and log file size), it would be easier to just record the field changes in some coded format (such as offset or field number and raw field data), rather than include the field name, type, and data. But in order for that to be practical, you'd need to rely on the MAP FILE at the time you searched/decoded the log file, not just at the time the events were captured. (This is something like the issue with runtime location counters, which are only useful if you have an LSX matching the RUN file.)

As a practical matter, record layouts are not likely to change very often, so this may be a reasonable condition/limitation to impose. But it does detract slightly from the ease of use.

Here's the initial beta release (which doesn't include any of the field-level ideas discussed immediately above)...

ash65notes.txt
ash-6.5.1642.0-w32-upd.zip
ash-6.5.1642.0-w32c-upd.zip
ash-6.5.1642.0-el7-upd.tz
ash-6.5.1642.0-el5-upd.tz

To experiment with the new hook mechanism, pull down the latest SOSLIB (primarily the 907010 and 907016 directories to update the SOSFUNC: and ASHINC: files) and then compile and run FHOOKTST4.BP from 908050 of the EXLIB .

Neat, I shell give it a bash when I get few seconds

Downloaded and had quick play with FHOOKTST4 and it does exactly what it says on the tin I'll play more later got side tracked with some SQL/Ashell thing.