15 Jul
2014
15 Jul
'14
11:43 p.m.
Yes, an evil necessary to get things going.
The very definition of original sin.
Doug
Larry McVoy wrote:
>>> For stdio, of course, one would need
fsplice(3), which must flush the
>>> in-process buffers--penance for stdio's original sin of said buffering.
>> Err, why is buffering data in the process a
sin? (Or was this just a
>> humourous aside?)
> Process A spawns process B, which reads stdin with
buffering. B gets
> all it deserves from stdin and exits. What's left in the buffer,
> intehded for A, is lost. Sinful.
It really depends on what you want. That buffering is
a big win for
some use cases. Even on today's processors reading a byte at a time via
read(2) is costly. Like 5000x more costly on the laptop I'm typing on:
16 Jul
16 Jul
12:32 a.m.
New subject: the sin of buffering [offshoot of excise process from a pipeline]
I dunno, we have a distributed source management system that uses a lot
of network I/O. We've carefully layered stdio on top of it because we
had many cases where it was a performance bummer.
Personally, I've come to really love stdio, at least our version of it.
Want your stream compressed or uncompressed?
fpush(&stdin, fopen_vzip(stdin, "r"));
Want your stream integrity checked with a CRC per block and an XOR block
at the end so you can correct any single block error?
fpush(&stdout, fopen_crc(stdout, "w", 0, 0));
I'm a performance guy for the most part and while read/write seem like
the fastest way to move stuff around that's only true for really nicely
formed data, page sized blocks or bigger. Fine for benchmarking but if
you want to approach that performance with poorly formed data, like
small blocks, different sized blocks, that buffering layer smooths
things out. You pay an extra bcopy() but that's typically lost in
the noise.
I used to hate the idea of stdio but working in real world applications
where I can't control the size of the data coming at me, yeah, I've
come to love stdio. It's pretty darn useful.
On Tue, Jul 15, 2014 at 07:43:49PM -0400, Doug McIlroy wrote:
Yes, an evil necessary to get things going.
The very definition of original sin.
Doug
Larry McVoy wrote:
--
---
Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm
>>> For stdio, of course, one would need
fsplice(3), which must flush the
>>> in-process buffers--penance for stdio's original sin of said buffering.
>> Err, why is buffering data in the process
a sin? (Or was this just a
>> humourous aside?)
> Process A spawns process B, which reads stdin
with buffering. B gets
> all it deserves from stdin and exits. What's left in the buffer,
> intehded for A, is lost. Sinful.
> It really depends on what you want. That buffering is a big win for
> some use cases. Even on today's processors reading a byte at a time via
> read(2) is costly. Like 5000x more costly on the laptop I'm typing on: 
3:53 a.m.
New subject: the sin of buffering [offshoot of excise process from a pipeline]
Larry McVoy scripsit:
Want your stream compressed or uncompressed?
fpush(&stdin, fopen_vzip(stdin, "r"));
Me, I would have done it with freopen(stdin, "rv").
--
John Cowan http://www.ccil.org/~cowan cowan(a)ccil.org
Si hoc legere scis, nimium eruditionis habes.
4:05 a.m.
New subject: the sin of buffering [offshoot of excise process from a pipeline]
On Tue, Jul 15, 2014 at 11:53:03PM -0400, John Cowan wrote:
Larry McVoy scripsit:
We tried that but the problem is that you can't encode all the options you
want in just a character. Compression doesn't take options, the CRC/XOR
layer wants to know how big you might think the file is (because we
support blocksizes from about 256B to 256K and we want to know the
file size to guess the block size).
--
---
Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm
Want your stream compressed or uncompressed?
fpush(&stdin, fopen_vzip(stdin, "r"));
Me, I would have done it with freopen(stdin, "rv").
6:03 a.m.
New subject: the sin of buffering [offshoot of excise process from a pipeline]
Larry McVoy scripsit:
We tried that but the problem is that you can't
encode all the options you
want in just a character. Compression doesn't take options, the CRC/XOR
layer wants to know how big you might think the file is (because we
support blocksizes from about 256B to 256K and we want to know the
file size to guess the block size).
It's a string: you can have as many characters as you want.
--
John Cowan http://www.ccil.org/~cowan cowan(a)ccil.org
Dievas dave dantis; Dievas duos duonos --Lithuanian proverb
Deus dedit dentes; deus dabit panem --Latin version thereof
Deity donated dentition;
deity'll donate doughnuts --English version by Muke Tever
God gave gums; God'll give granary --Version by Mat McVeagh
2:30 p.m.
New subject: the sin of buffering [offshoot of excise process from a pipeline]
On Wed, Jul 16, 2014 at 02:03:58AM -0400, John Cowan wrote:
Larry McVoy scripsit:
I understand your desire to have one API. We tried and it just wasn't
practical. Imagine pushing an encryption layer that wants a key,
XOR layer that wants block size, etc.
We tried that but the problem is that you
can't encode all the options you
want in just a character. Compression doesn't take options, the CRC/XOR
layer wants to know how big you might think the file is (because we
support blocksizes from about 256B to 256K and we want to know the
file size to guess the block size).
It's a string: you can have as many characters as you want. 
2:56 p.m.
New subject: the sin of buffering [offshoot of excise process from a pipeline]
Why can't those be embedded in the relevant string? freopen(fp,
"rx{128}")
or something?
On Wed, Jul 16, 2014 at 10:30 AM, Larry McVoy <lm(a)mcvoy.com> wrote:
On Wed, Jul 16, 2014 at 02:03:58AM -0400, John Cowan
wrote:
I understand your desire to have one API. We tried and it just wasn't
practical. Imagine pushing an encryption layer that wants a key,
XOR layer that wants block size, etc.
_______________________________________________
TUHS mailing list
TUHS(a)minnie.tuhs.org
https://minnie.tuhs.org/mailman/listinfo/tuhs
Larry McVoy scripsit:
> We tried that but the problem is that you can't encode all the options
you
> want in just a character. Compression
doesn't take options, the
CRC/XOR
layer
wants to know how big you might think the file is (because we
support blocksizes from about 256B to 256K and we want to know the
file size to guess the block size).
It's a string: you can have as many characters as you want.
3:41 p.m.
New subject: the sin of buffering [offshoot of excise process from a pipeline]
What is being provided is a generic layering system on top of stdio.
Any sort of conversion you want. Encryption, CRC, XOR block, compression
(we support gzip and lz4). Those are just the layers we use right now,
it's easy to imagine others being added.
The point is that there is no one API that is going to pleasantly encode
all of the options to all of those layers and any that may come later.
Are you seriously suggesting that you want to read the freopen(3)
man page and see all of these options explained? That's the classic
open source way, dump everything in one poorly thought out man page.
It's not the Unix way, people think about it harder.
For the record, I pushed for the single string encoding as well but got
pushed off it as I realized the API wasn't as simple as I imagined.
While you could do it that way you shouldn't do it that way, it's just
not a good API.
I'm very pleased with how it turned out in our code, other than a handful
of fpush() calls, it just looks like stock stdio.
On Wed, Jul 16, 2014 at 10:56:57AM -0400, Dan Cross wrote:
Why can't those be embedded in the relevant
string? freopen(fp, "rx{128}")
or something?
On Wed, Jul 16, 2014 at 10:30 AM, Larry McVoy <lm(a)mcvoy.com> wrote:
> On Wed, Jul 16, 2014 at 02:03:58AM -0400, John Cowan wrote:
> > Larry McVoy scripsit:
> >
> > > We tried that but the problem is that you can't encode all the
options
> you
> > > want in just a character. Compression doesn't take options, the
> CRC/XOR
> > > layer wants to know how big you might think the file is (because we
> > > support blocksizes from about 256B to 256K and we want to know the
> > > file size to guess the block size).
> >
> > It's a string: you can have as many characters as you want.
>
> I understand your desire to have one API. We tried and it just wasn't
> practical. Imagine pushing an encryption layer that wants a key,
> XOR layer that wants block size, etc.
> _______________________________________________
> TUHS mailing list
> TUHS(a)minnie.tuhs.org
> https://minnie.tuhs.org/mailman/listinfo/tuhs
>
--
---
Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm
3581
days inactive
3582
days old
7 comments
4 participants
participants (4)
-
Dan Cross -
Doug McIlroy -
John Cowan -
Larry McVoy