Right... that's one of the reasons we created AST's and tried to get them in the POSIX spec and an alternative to signals. Instead of trying to 'fix' them, we decided to inject a new scheme with better semantics (like being queued, prioritized, guaranteed to be delivered). They were *.4 for a while and got replaced with sigqueue which sort helps but does not solve the issues. I'm always torn, do you add a new interface and risk bloat, or try to extend the old. We never did it in RTU, but talked about it for Stellix (and never did it there either), but we toyed with making AST's the kernel primitive and then trying to build signals from them in a user space library -> which at the time, we thought you can do. Although there are some strange side effects of signals that you would have really think through. They have been extended again with POSIX and SVR4 so I'm not so sure, It's hard because if I/O (like DMA) is in progress, what are the proper semantics to exit/roll back? When do you stop the transfer and what happens. When doing direct to/from disk (like a in a real-time system), this can get tricky. Traditional UNIX semantics were that when a DMA was started, the process is blocked at high priority until the I/O completes. Which is fine in the standard case, but begs the question of what happens when things go south. signals match an easy implementation of the V6 kernel. BSD did try to fix them a little, but we all know that caused as many issues as it solved. Clem On Fri, Dec 1, 2017 at 10:53 AM, Dan Cross <crossd(a)gmail.com> wrote:

On Fri, Dec 1, 2017 at 9:18 AM, Larry McVoy <lm(a)mcvoy.com> wrote: The I/O subsystem is great at knowing disks, pages and buffers. Lousy at knowing processes. It has no hooks into, well, anything apart from FOOstrategy (and if you are lucky first open and last close). And FOOstrategy is far removed from anything that the process knows about. It knows about vmpages and open file descriptors. You'd need to plumb something into the vnode code that could take a request for a signal. Then that signal would need to go down to the driver somehow. And you'd need to know what I/O was pending for pages that are in that process, and somehow have an ID to cancel just those requests. With the layering involved, it would be extremely tricky. Many of the io completion routines cause more I/O to happen, especially when swapping. Plus, modern disks have a hardware queue depth. There is no way to do say 'finish this and do no more' because there's stuff in the hardware queue. SCSI is sensible and you may be able to selectively cancel I/O transactions, but ATA is not sensible: you cancel the whole queue and cope with the fallout (usually by rescheduling the I/Os). Perhaps. Even that may have issues with cleanup because you may also need other pages to complete the I/O processing since I think that things like aio allocate a tiny bit of memory associated with the requesting process and need that memory to finish the I/O. It's certainly not a requirement that all I/O initiated by userland have no extra state stored in the process' address space associated with it. Since the unwinding happens at a layer higher than the disk driver, who knows what those guys do, eh? Warner

FYI, the way signals actually work is that they are delivered only by crossing through the user/kernel boundary. (That mostly even includes those that kill a process entirely, except that there's some optimizations there for some cases. If the CPU is running in user space for some *other* process, we have to arrange for the signalled process to be scheduled to run, or if it's on another CPU, take an interrupt so as to get into the kernel and hence cross that boundary.) This part is natural, since signal delivery means "run a function in user space, with user privileges". If you're currently *in* the kernel on behalf of some user process, that means you have to get *out* of it in that same process. This is why signals interrupt system calls, making them return with EINTR. To avoid having system calls that haven't really started -- have not done anything yet -- from returning EINTR, the BSD and POSIX SA_RESTART options work by just making the "resume" program counter address point back to the "make system call" instruction. Essentially: frame->pc -= instruction_size; This can't be done if the system call has actually done something, so read() or write() on a slow device like a tty will return a short count (not -1 and EINTR). The BSD implementation these days is to call the various *sleep kernel functions (msleep, tsleep, etc) with PCATCH "or"-ed into the priority to indicate that it is OK to have a signal interrupt the sleep. If so, the sleep call returns ERESTART (a special error code that the syscall handler notices and does the "subtract from frame->pc" trick). In the old days, PCATCH was implied by the sleep priority; all we did was make it an explicit flag, and do manual unwinding (the V6 kernel used the equivalent of longjmp to get to the EINTR-returner so that the main line code path did not have to check). The I/O subsystem generally calls *sleep without PCATCH, though. Chris

On Dec 1, 2017, at 9:26 AM, Larry McVoy <lm(a)mcvoy.com> wrote: So OOM code kills a (random) process in hopes of freeing up some pages but if this process is stuck in diskIO, nothing can be freed and everything grinds to a halt. Is this right? If so, one work around is to kill a process that is *not* sleeping at an uninterruptable priority :-)/2 This is separate from any policy of how to choose a victim. If the queue of dirty pages is growing longer and longer as the page washer can't keep up, this is analogous to the bufferbloat problem in networking. You have to test if this is what is going on. If so, may be you can figure out how to keep the queues short. But before any fixes, I would strongly suggest instrumenting the code to understand what is going on and then instrument the code further to test out various hypotheses. Once a clear mental model is in place, the fix will be obvious!

Hi Larry, Is that because the pages have been dirty for so long they've reached the VM-writeback timeout even though there's no pressure to use them for something else? Or has that been lengthened because you don't fear power loss wiping volatile RAM? -- Cheers, Ralph. https://plus.google.com/+RalphCorderoy

On Fri, 01 Dec 2017 15:09:34 -0800 Larry McVoy <lm(a)mcvoy.com> wrote: Larry McVoy writes: How many times do processes walk their memory before this condition occurs? So what may be happening is that a process references a page, it page faults, the kernel finds its phys page has been paged out, so it looks for a free page and once a free page is found, the process will block on page in. Or if there is no free page, it has to wait until some other dirty page is paged out (but this would be a different wait queue). As more and more processes do this, the system runs out of all free pages. Can you find out how many processes are waiting under what conditions, how long they wait and how these queue lengths are changing over time? You can use a ring buffer to capture last 2^N measurements and dump them in the debugger when everything grinds to a halt. The OS equivalent of RED (random early drop) would be if a process kills itself. e.g. when some critical metric crosses a highwater mark. Another option would be to return with an EFAULT and the process can either kill itself or free up the page or something. [I have used EFAULT to dyanmically allocate *more* pages but no reason why the same can be used to free up memory!]

On Fri, 01 Dec 2017 16:48:50 -0800 Larry McVoy <lm(a)mcvoy.com> wrote: Larry McVoy writes: You said 10x25GB but you have 256GB. So there is still 6GB left... This confuses me. Before I make more false assumptions, can you show the code? Can you use kgdb to find out where they sleep? Won't kgdb tell you? Or you can insert printfs. You should also print something in your program as it walks a few pages and see if this happens at almost the same time (when all pages are used up). tty code probably assumes memory shortfall is a short term issue (which is likely). Not the case with your test (unless I misguessed). Now I am starting to think this happens as soon as all the phys pages are used up. But looking at your program would help. [removd cc: TUHS]

Larry McVoy <lm(a)mcvoy.com> wrote: Um, pardon me for asking the obvious question, but why not just configure a few gigs of swap to give the OS some breathing room? Most modern systems let you use a regular old file in the filesystem for swap space, instead of having to repartition your disk and using a dedicated partition. I'd be suprised if your *BSD box didn't let you do that too. It's a little slower, but a gazillion times more convenient. Just a thought, Arnold

On Mon, Dec 4, 2017 at 9:36 AM, <arnold(a)skeeve.com> wrote: The deployed systems in the field have swap space configured. But if we're not careful we can still run out. Larry's tests are at the extreme limits, to be sure, and not having swap exacerbates the bad behavior at the limit. If he's trying to ameliorate bad effects, doing it on worst case scenario certainly helps. The BSD box allows for it, and it's not that much slower, but it kinda misses the point of the exercise. Also, the systems in questions often operate at the limits of the disk subsystem, so additional write traffic is undesirable. They are also SSDs, so doubly undesirable where avoidable. Warner

On Mon, 4 Dec 2017, arnold(a)skeeve.com wrote: Doesn't it have to be a contiguous file (for DMA), or is scatter/gather now supported? -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer."

Pages are pages. The filesystem handles the details for the offset of each one. There's no contiguous on disk requirement. Warner On Mon, Dec 4, 2017 at 3:54 PM, Ron Natalie <ron(a)ronnatalie.com> wrote:

On 12/4/2017 7:49 PM, Dave Horsfall wrote: Nope, unless mkfile is is guaranteed to give you contiguous space: # uname -a SunOS vmsol8 5.8 Generic_108529-05 i86pc i386 i86pc # mkfile 100m a.a # swap -a /a.a # swap -l swapfile             dev  swaplo blocks   free /dev/dsk/c0d0s1     102,1       8 1048936 1048896 /a.a                  -        8 204792 204792

I can not speak for others but both Masscomp’s Real Time Unix and Stellar’s Stellix supported contiguous files and could swap to them. And yes they were preallocated at creat(2) time. Sent from my PDP-7 Running UNIX V0 expect things to be almost but not quite.