From athornton at gmail.com Sun Aug 1 00:21:01 2021 From: athornton at gmail.com (Adam Thornton) Date: Sat, 31 Jul 2021 07:21:01 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <20210731142533.69caf929@moon> References: <20210731142533.69caf929@moon> Message-ID: On Jul 31, 2021, at 5:25 AM, Michael Siegel wrote: While doing that, I learned that there is a better way to approach this problem – beyond using getopt(s) (which never really made sense to me) and having to write case statements in loops every time: Define a grammar, let a pre-built parser do the work, and have the parser provide the results to the program. I see that Dan Halbert beat me to mentioning "click." The trick with shell is that unless you write the parser in shell, which is going to be miserable, you’re doing it in a command in a subshell, and therefore your return values have to be a structured stream of bytes on stdout, which the parent shell is going to have to interpret. An eval-able shell fragment, where you have a convention of what the variables you get from the option parser will be, is probably the easiest way, since from the parent that would look like: $(parse_my_opts $*) # Magic variables spring to life if [ “$OPT_SUBCOMMAND_0” == “burninate” ]; then …. Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From athornton at gmail.com Sun Aug 1 00:25:24 2021 From: athornton at gmail.com (Adam Thornton) Date: Sat, 31 Jul 2021 07:25:24 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: Digressing a bit (but only a bit) talking about IPC: Powershell and CMS PIPELINES both take the approach of more structured pipelines, where pipe contents are not just streams of bytes but can be structured records. This offers a lot of power, but it also inhibits the ability to arbitrarily compose pipe stages, because you've effectively introduced a type system. On the other hand you can certainly argue that stream-of-bytes pipes ALSO introduce a type system, it's just a completely ad-hoc, undocumented, and fragile one that relies on the cooperation of both ends of the pipe to work at all, and you'd be right. In practice...well, I'd rather use stream-of-bytes, but I am more comfortable in Unix-like environments than Powershell, and my CMS PIPELINES skills are quite rusty now. On Sat, Jul 31, 2021 at 7:21 AM Adam Thornton wrote: > > > On Jul 31, 2021, at 5:25 AM, Michael Siegel wrote: > > While doing that, I learned that there is a better way to approach > this problem – beyond using getopt(s) (which never really made sense to > me) and having to write case statements in loops every time: Define a > grammar, let a pre-built parser do the work, and have the parser > provide the results to the program. > > > I see that Dan Halbert beat me to mentioning "click." > > The trick with shell is that unless you write the parser in shell, which > is going to be miserable, you’re doing it in a command in a subshell, and > therefore your return values have to be a structured stream of bytes on > stdout, which the parent shell is going to have to interpret. An eval-able > shell fragment, where you have a convention of what the variables you get > from the option parser will be, is probably the easiest way, since from the > parent that would look like: > > $(parse_my_opts $*) > # Magic variables spring to life > if [ “$OPT_SUBCOMMAND_0” == “burninate” ]; then …. > > Adam > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rich.salz at gmail.com Sun Aug 1 01:45:52 2021 From: rich.salz at gmail.com (Richard Salz) Date: Sat, 31 Jul 2021 11:45:52 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <20210731142533.69caf929@moon> References: <20210731142533.69caf929@moon> Message-ID: Look for "comnd jsys" that exact spelling. Source code is around. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.winalski at gmail.com Sun Aug 1 01:56:51 2021 From: paul.winalski at gmail.com (Paul Winalski) Date: Sat, 31 Jul 2021 11:56:51 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <20210731142533.69caf929@moon> References: <20210731142533.69caf929@moon> Message-ID: On 7/31/21, Michael Siegel wrote: > > While doing that, I learned that there is a better way to approach > this problem – beyond using getopt(s) (which never really made sense to > me) and having to write case statements in loops every time: Define a > grammar, let a pre-built parser do the work, and have the parser > provide the results to the program. This method for handling command lines dates back at least to the 1970s. The COMND JSYS (system call) in TOPS-20 operated this way, as does the DCL command line interface in OpenVMS. As you pointed out it can greatly simplify the code in the application. It also permits command completion. If the command has a long-winded option, such as -supercalifragilisticexpialidocious, I can type -super then hit the TAB key and as long as there is only one option that starts with -super the parser will fill in the rest of the long keyword. It also means that you can provide interactive help. At any point the user can type a question mark and the command interpreter will say what syntactic element is expected next. The TOPS-20 COMND JSYS implemented both of these features, and I think that command completion was eventually added to the VMS command interpreter, too. This method of command line parsing also enforces a degree of uniformity of syntax between the command lines of the various utilities and applications. -Paul W. From clemc at ccc.com Sun Aug 1 02:03:13 2021 From: clemc at ccc.com (Clem Cole) Date: Sat, 31 Jul 2021 12:03:13 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: https://github.com/PDP-10/sri-nic/blob/master/files/fs/c/ccmd/ccmdmd.unx On Sat, Jul 31, 2021 at 11:46 AM Richard Salz wrote: > Look for "comnd jsys" that exact spelling. Source code is around. > > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rich.salz at gmail.com Sun Aug 1 02:06:14 2021 From: rich.salz at gmail.com (Richard Salz) Date: Sat, 31 Jul 2021 12:06:14 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: You gave it away :) Half the fun is getting the search right. The old kermit program from Columbia has an implementation in portable (for its time) C. On Sat, Jul 31, 2021 at 12:03 PM Clem Cole wrote: > https://github.com/PDP-10/sri-nic/blob/master/files/fs/c/ccmd/ccmdmd.unx > > On Sat, Jul 31, 2021 at 11:46 AM Richard Salz wrote: > >> Look for "comnd jsys" that exact spelling. Source code is around. >> >> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sun Aug 1 02:17:18 2021 From: clemc at ccc.com (Clem Cole) Date: Sat, 31 Jul 2021 12:17:18 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: Sorry, hit return too soon. I remember an old AAUGN newsletter describing it. If I recall it was original done for kermit. The same idea is in tcsh also. Which came first, I don't remember. Cut/pasted from AAUGN Vol8 # 2 ----------------- CCMD: A Version of COMND in C *Andrew Lowry* *Howard Kaye * Columbia University CCMD is a general parsing mechanism for developing User Interfaces to programs. It is based on the functionality of TOP5.20's COMND Jsys. CCMD allows a program to parse for various field types (file names, user names, dates and times, keywords, numbers, arbitrary text, tokens, *etc*.). It is meant to supply a homogeneous user interface across a variety of machines and operating systems for C programs. It currently runs under System V UNIX, 4.2/4.3 BSD, Ultrix 1.2/2.0, and MSDOS. The library defines various default actions (user settable), and allows field completion, help, file indirection, comments, *etc*. on a per field basis. Future plans include command line editing, command history, and ports to other operating systems (such as VMS). CCMD is available for anonymous FTP from [CU20B.COLUMBIA.EDU]WS:*.* For further information, send mail to: info-ccmd-request at cu20b.columbia.edu seismo!columbia!cunixc!info-ccmd-request On Sat, Jul 31, 2021 at 12:03 PM Clem Cole wrote: > https://github.com/PDP-10/sri-nic/blob/master/files/fs/c/ccmd/ccmdmd.unx > > On Sat, Jul 31, 2021 at 11:46 AM Richard Salz wrote: > >> Look for "comnd jsys" that exact spelling. Source code is around. >> >> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From crossd at gmail.com Sun Aug 1 02:19:27 2021 From: crossd at gmail.com (Dan Cross) Date: Sat, 31 Jul 2021 12:19:27 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: On Sat, Jul 31, 2021 at 11:57 AM Paul Winalski wrote: > On 7/31/21, Michael Siegel wrote: > > > > While doing that, I learned that there is a better way to approach > > this problem – beyond using getopt(s) (which never really made sense to > > me) and having to write case statements in loops every time: Define a > > grammar, let a pre-built parser do the work, and have the parser > > provide the results to the program. > > This method for handling command lines dates back at least to the > 1970s. The COMND JSYS (system call) in TOPS-20 operated this way, as > does the DCL command line interface in OpenVMS. As you pointed out it > can greatly simplify the code in the application. It also permits > command completion. If the command has a long-winded option, such as > -supercalifragilisticexpialidocious, I can type -super then hit the > TAB key and as long as there is only one option that starts with > -super the parser will fill in the rest of the long keyword. It also > means that you can provide interactive help. At any point the user > can type a question mark and the command interpreter will say what > syntactic element is expected next. The TOPS-20 COMND JSYS > implemented both of these features, and I think that command > completion was eventually added to the VMS command interpreter, too. > > This method of command line parsing also enforces a degree of > uniformity of syntax between the command lines of the various > utilities and applications. > There was someone posting here on TUHS a while back about leveraging a special context-sensitive `--shell-help` or similar command line program and synthesizing a protocol between the shell and a program to provide TOPS-20 like command completion. It was nowhere near what you get from the COMND JSYS, but seemed like a reasonable approximation. This is verging on COFF territory, but one of the reasons such a mechanism is unlike what you get from TOPS-20 is that, in that system, as soon as you type the name of a command, you're effectively running that command; the process model is quite different from that of Unix. With respect to command line handling in general, I think there are some attempts at making things more rational available in modern languages. Command line parsing packages for Go and the `clap` package for Rust come to mind ( https://rust-lang-nursery.github.io/rust-cookbook/cli/arguments.html). I've used clap recently in a few places and it's very convenient. - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sun Aug 1 02:21:07 2021 From: clemc at ccc.com (Clem Cole) Date: Sat, 31 Jul 2021 12:21:07 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: Sorry, I remembered it from AUUGN and checked those first, then searched for "CCMD Unix Columbia.edu" and got the hit on Lar's PDP-10 sources. and note the dyslexic spelling in my earlier email of AUUGN -- sigh. On Sat, Jul 31, 2021 at 12:06 PM Richard Salz wrote: > You gave it away :) Half the fun is getting the search right. > > The old kermit program from Columbia has an implementation in portable > (for its time) C. > > On Sat, Jul 31, 2021 at 12:03 PM Clem Cole wrote: > >> https://github.com/PDP-10/sri-nic/blob/master/files/fs/c/ccmd/ccmdmd.unx >> >> On Sat, Jul 31, 2021 at 11:46 AM Richard Salz >> wrote: >> >>> Look for "comnd jsys" that exact spelling. Source code is around. >>> >>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From beebe at math.utah.edu Sun Aug 1 02:27:00 2021 From: beebe at math.utah.edu (Nelson H. F. Beebe) Date: Sat, 31 Jul 2021 10:27:00 -0600 Subject: [TUHS] Systematic approach to command-line interfaces Message-ID: Besides C-Kermit on Unix systems, the TOPS-20 command interface is used inside the mm mail client, which I've been using for decades on TOPS-20, VMS, and several flavors of Unix: http://www.math.utah.edu/pub/mm mm doesn't handle attachments, or do fancy display of HTML, and thus, cannot do anything nasty in response to incoming mail messages. I rarely need to extract an attachment, and I then save the message in a temporary file and run munpack on it. Here are some small snippets of its inline help: MM] read (messages) ? message number or range of message numbers, n:m or range of message numbers, n-m or range of message numbers, n+m (m messages beginning with n) or "." to specify the current message or "*" to specify the last message or message sequence, one of the following: after all answered before current deleted flagged from inverse keyword last longer new on previous-sequence recent seen shorter since subject text to unanswered undeleted unflagged unkeyword unseen or "," and another message sequence R] read (messages) flagged since yesterday [message(s) appear here] MM] headers (messages) since monday longer (than) 100000 [list of long messages here] ------------------------------------------------------------------------------- - Nelson H. F. Beebe Tel: +1 801 581 5254 - - University of Utah FAX: +1 801 581 4148 - - Department of Mathematics, 110 LCB Internet e-mail: beebe at math.utah.edu - - 155 S 1400 E RM 233 beebe at acm.org beebe at computer.org - - Salt Lake City, UT 84112-0090, USA URL: http://www.math.utah.edu/~beebe/ - ------------------------------------------------------------------------------- From crossd at gmail.com Sun Aug 1 02:30:33 2021 From: crossd at gmail.com (Dan Cross) Date: Sat, 31 Jul 2021 12:30:33 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: On Sat, Jul 31, 2021 at 12:18 PM Clem Cole wrote: > Sorry, hit return too soon. I remember an old AAUGN newsletter > describing it. If I recall it was original done for kermit. The same > idea is in tcsh also. Which came first, I don't remember. Cut/pasted from > AAUGN Vol8 # 2 > Frank da Cruz wrote a very nice reminiscence of the DECSYSTEM-20s at Columbia that discusses the creation of CCMD as they decommissioned the PDP-10s and switched to Unix on VAXen (and then Suns). http://www.columbia.edu/kermit/dec20.html When I was a student, we were still given accounts on the CUNIX cluster; 64-bit SPARC machines running Solaris at the time. At the time, the actress Julia Styles was a student. One day, I was walking out of Mudd (the engineering building) with a friend of mine who suddenly grabbed my arm and said, "oh my god oh my god oh my god that's Julia Styles!" Being perpetually ignorant of popular culture, I had no idea who she was referring to confusedly thought she meant Julia Child, the late host of a cooking show. "...But I thought she was dead?" "No, Dan, that's Julia Child!" We decided to look up Ms Styles in the student directory, but being a celebrity she wasn't listed. However, one could still discover her "UNI" (login name) by grepping for her in the NIS password database. We did that and sent her an email: "Christy was too embarrassed to say hi to you and Dan thought you were Julia Child." Predictably, she did not respond. In retrospect, I idly wonder how many such emails she got, most presumably of the creepy variety, but we just thought ours was funny. It appears that CUNIX still exists: https://cuit.columbia.edu/unix - Dan C. ----------------- > CCMD: A Version of COMND in C > > *Andrew Lowry* > *Howard Kaye * > > Columbia University > > CCMD is a general parsing mechanism for developing User Interfaces to > programs. It is based on the functionality of TOP5.20's COMND Jsys. CCMD > allows a program to parse for various field types (file names, user names, > dates and times, keywords, numbers, arbitrary text, tokens, *etc*.). It > is meant to supply a homogeneous user interface across a variety of > machines and operating systems for C programs. It currently runs under > System V UNIX, 4.2/4.3 BSD, Ultrix 1.2/2.0, and MSDOS. The library defines > various default actions (user settable), and allows field completion, help, > file indirection, comments, *etc*. on a per field basis. Future plans > include command line editing, command history, and ports to other operating > systems (such as VMS). > > CCMD is available for anonymous FTP from > [CU20B.COLUMBIA.EDU]WS:*.* > > For further information, send mail to: > > info-ccmd-request at cu20b.columbia.edu > seismo!columbia!cunixc!info-ccmd-request > > > > On Sat, Jul 31, 2021 at 12:03 PM Clem Cole wrote: > >> https://github.com/PDP-10/sri-nic/blob/master/files/fs/c/ccmd/ccmdmd.unx >> >> On Sat, Jul 31, 2021 at 11:46 AM Richard Salz >> wrote: >> >>> Look for "comnd jsys" that exact spelling. Source code is around. >>> >>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Sun Aug 1 02:41:04 2021 From: clemc at ccc.com (Clem Cole) Date: Sat, 31 Jul 2021 12:41:04 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <20210731142533.69caf929@moon> References: <20210731142533.69caf929@moon> Message-ID: On Sat, Jul 31, 2021 at 8:36 AM Michael Siegel wrote: > Hello, > > I've recently started to implement a set of helper functions and > procedures for parsing Unix-like command-line interfaces (i.e., POSIX + > GNU-style long options, in this case) As an old guy, I am amused to read these words .. because UNIX did not have a command-line parsing standard., and I remember the wars. If you came from a system, where the program was exec's with the command line parameters pre-parsed (like the DEC world, ITS, and some others); UNIX seemed foreign and often consider 'bad' by folks. The biggest argument (which was reasonable) was Unix command, sometimes used 'keys' (like tp/tar and the like) and others used switches (cp, ed). Folks new to UNIX often b*tched as it being 'inconsistent (read things like the 'UNIX Haters Book'). I admit I was 'surprised' when I came there in the Fifth Edition in the mid-70s from the PDP-10 world, but as a programmer, I ended up really liking the fact that the command-line was not pre-parsed, other than white space removal and I did not have figure out some strange syntax for findnext() and other UUO/JSYS from my previous life. So by the late 70's early 80's, a number of different UNIX parsing schemes popped up. Like the stuff from Columbia Richard pointed out. TCL in some ways end result, which had a life that was useful, but fell away too eventually. The whole getopt(3) thing appeared originally inside of BTL. The first version I was was from USB (Summit), but I'm not sure they were the original authors. One problem was that it was tied up with later AT&T licenses [i.e. PWB or later] and was not in Research, the USENIX community lacked it. Thus when AT&T brought it to us to consider for POSIX.2, there was balking. The ISV's seemed to like it, but there was not a lot of support elsewhere. At some point, somebody in the USENIX community wrote a version and posted it to comp.unix.sources and some people began to use it. Of course, GNU had to take it and pee on it, so we got the long file name stuff. All in all, it's what's you are used I suspect. The AT&T whole getopt(3) thing works (I can deal with keys too BTW). I guess I just don't get excited about it, these days. Clem -------------- next part -------------- An HTML attachment was scrubbed... URL: From rly1 at embarqmail.com Sun Aug 1 02:47:26 2021 From: rly1 at embarqmail.com (Ron Young) Date: Sat, 31 Jul 2021 09:47:26 -0700 Subject: [TUHS] Systematic approach to command-line interfaces Message-ID: > On 7/31/21, Michael Siegel wrote: > The TOPS-20 COMND JSYS implemented both of these features, and I think that command > completion was eventually added to the VMS command interpreter, too. FYI, There is also a unix version of the COMND JSYS capability. It was developed at Columbia University as part of their "mm" mail manager. It is located in to the ccmd subdirectory in the mm.tar.gz file. url: https://www.kermitproject.org/mm/ ftp://ftp.kermitproject.org/kermit/mm/mm.tar.gz -ron From ality at pbrane.org Sun Aug 1 03:30:18 2021 From: ality at pbrane.org (Anthony Martin) Date: Sat, 31 Jul 2021 10:30:18 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <20210731142533.69caf929@moon> References: <20210731142533.69caf929@moon> Message-ID: Michael Siegel once said: > So, I've prepared a bit of a write-up, pondering on the pros and cons > of two different ways of having task-specific tool sets > (non-hierarchical command sets vs. sub-commands) that is available at > > https://www.msiism.org/files/doc/unix-like_command-line_interfaces.html > > I tend to think the sub-command approach is better. But I'm neither a UI > nor a Unix expert and have no formal training in computer things. So, I > thought this would be a good place to ask for comment (and get some > historical perspective). You're missing the approach taken in Plan 9 (and 10th edition Unix): put related commands in a directory and use a shell that doesn't restrict the first argument of a command to a single path element. This lets you execute commands like: auth/as disk/prep git/rebase ip/ping ndb/dns upas/send without having a prefix on every command name or single large binaries with every command linked in as subcommands. Cheers, Anthony From cowan at ccil.org Sun Aug 1 03:41:55 2021 From: cowan at ccil.org (John Cowan) Date: Sat, 31 Jul 2021 13:41:55 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: On Sat, Jul 31, 2021 at 12:41 PM Clem Cole wrote: The biggest argument (which was reasonable) was Unix command, sometimes > used 'keys' (like tp/tar and the like) and others used switches (cp, ed). > That's modest (especially now that tar accepts "-" before the key) compared to remembering when to use -F, when to use -d, and when to use -t to specify the field separator, and when you are stuck without a field separator option; also what the default is with no option (I'd prefer "arbitrary amount whitespace" in all cases, but that's often not available at all). These inconsistencies still piss me off no end. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cowan at ccil.org Sun Aug 1 03:46:28 2021 From: cowan at ccil.org (John Cowan) Date: Sat, 31 Jul 2021 13:46:28 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: On Sat, Jul 31, 2021 at 1:39 PM Anthony Martin wrote: > You're missing the approach taken in Plan 9 (and > 10th edition Unix): put related commands in a > directory and use a shell that doesn't restrict > the first argument of a command to a single path > element. > What that doesn't give you is the ability to say "git diff ", which is very nice and makes the inconsistencies I just posted on less likely. Fortunately, any getopt-variant can deal with these; you just have to pass the tail of argv and a suitably reduced value for argc to another call of the options parser. -------------- next part -------------- An HTML attachment was scrubbed... URL: From msi at malbolge.net Sun Aug 1 04:56:09 2021 From: msi at malbolge.net (Michael Siegel) Date: Sat, 31 Jul 2021 20:56:09 +0200 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: <20210731205609.07e8149f@moon> Am Sat, 31 Jul 2021 10:30:18 -0700 schrieb Anthony Martin : > Michael Siegel once said: > > So, I've prepared a bit of a write-up, pondering on the pros and > > cons of two different ways of having task-specific tool sets > > (non-hierarchical command sets vs. sub-commands) that is available > > at > > > > https://www.msiism.org/files/doc/unix-like_command-line_interfaces.html > > > > I tend to think the sub-command approach is better. But I'm neither > > a UI nor a Unix expert and have no formal training in computer > > things. So, I thought this would be a good place to ask for comment > > (and get some historical perspective). > > You're missing the approach taken in Plan 9 (and > 10th edition Unix): put related commands in a > directory and use a shell that doesn't restrict > the first argument of a command to a single path > element. > > This lets you execute commands like: > > auth/as > disk/prep > git/rebase > ip/ping > ndb/dns > upas/send > > without having a prefix on every command name or > single large binaries with every command linked > in as subcommands. Thanks for pointing this out. I had no idea. Unfortunately(?), I'm looking to make my life easier on more "Unix-like Unix-like systems" (for want of a better term), for the time being (Linux, BSD, maybe illumos). (I mean, which shell would I use to accomplish this on Unix?) And, as has already been pointed out, this approach doesn't allow for global command options before sub-commands, which pretty much defeats the sub-command approach altogether UI-wise, I'd say. Unrelated: I'm still having some technical difficulties with this list, namely that I don't receive any mail sent to it. (I'm using the Web archive to keep track of what's happening.) So, for me to be able to reply to a particular message, it would also have to be sent directly to me. Sorry for the inconvenience. The problem is already being worked on. -- Michael From jon at fourwinds.com Sun Aug 1 05:20:02 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Sat, 31 Jul 2021 12:20:02 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <20210731142533.69caf929@moon> References: <20210731142533.69caf929@moon> Message-ID: <202107311920.16VJK2jT2362871@fourwinds.com> Michael Siegel writes: > Hello, > > I've recently started to implement a set of helper functions and > procedures for parsing Unix-like command-line interfaces (i.e., POSIX + > GNU-style long options, in this case) in Ada. > > While doing that, I learned that there is a better way to approach > this problem – beyond using getopt(s) (which never really made sense to > me) and having to write case statements in loops every time: Define a > grammar, let a pre-built parser do the work, and have the parser > provide the results to the program. > > Now, defining such a grammar requires a thoroughly systematic approach > to the design of command-line interfaces. One problem with that is > whether that grammar should allow for sub-commands. And that leads to > the question of how task-specific tool sets should be designed. These > seem to be a relatively new phenomenon in Unix-like systems that POSIX > doesn't say anything about, as far as I can see. > > So, I've prepared a bit of a write-up, pondering on the pros and cons > of two different ways of having task-specific tool sets > (non-hierarchical command sets vs. sub-commands) that is available at > > https://www.msiism.org/files/doc/unix-like_command-line_interfaces.html > > I tend to think the sub-command approach is better. But I'm neither a UI > nor a Unix expert and have no formal training in computer things. So, I > thought this would be a good place to ask for comment (and get some > historical perspective). > > This is all just my pro-hobbyist attempt to make some people's lives > easier, especially mine. I mean, currently, the "Unix" command line is > quite a zoo, and not in a positive sense. Also, the number of > well-thought-out command-line interfaces doesn't seem to be a growing > one. But I guess that could be changed by providing truly easy ways to > make good interfaces. > > > -- > Michael Well, don't let me discourage you from doing what you want. But, in my opinion, it doesn't add value to do something that's already been done but differently; it detracts from value because now there's yet another competing way to do something. I'm actually surprised that the format of commands was as consistent as it was for as long as it was. Sure, I never liked the way that things like tar were inconsistent, but it was mostly good for a long time. I see two reasons for the more recent messiness. o Minimal learning of history by newer practicioners resulting in doing things in a way that they're familiar instead of learning the behavior of the target environment and fitting in. It's what I call the "Ugly American Tourist" model; I'm in your country so you should speak English; why would I learn your language? o The endless pile of --gnu-long-option-names. On one hand, I sort of understand the long option names because so many commands have so many options that there's no way to have one-character mnemonics. But, I would make the argument that if one has that many options it's a sign that the command is trying to do too much stuff. For example, I don't care for the compression options on tar. These are actually just invoking separate programs in a pipeline, which to me is the job of the shell. Sure, it can be convenient, but again, one of the advantages of the shell is that I can make scripts for anything that I'm doing so often that the convenience would be nice. And without restarting an old flame war, separate programs are more stylisticly in line and more useful than many of the cat options. So I never got getopt(). One of my rules is that I don't use a library in cases where the number of lines of gunk that that it takes to use a library function is >= the number of lines to just write it myself. Yeah, I know the "but the library has more eyeballs and is debugged" argument but in reality libraries are the source of many bugs. I've always taken the approach that I would never hire someone who had to use a library to implement a singly-linked list. Jon From clemc at ccc.com Sun Aug 1 05:41:17 2021 From: clemc at ccc.com (Clem Cole) Date: Sat, 31 Jul 2021 15:41:17 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <20210731205609.07e8149f@moon> References: <20210731142533.69caf929@moon> <20210731205609.07e8149f@moon> Message-ID: On Sat, Jul 31, 2021 at 2:58 PM Michael Siegel wrote: > I mean, which shell would I use to accomplish this on Unix? In the old days, when the first Unix shell wars started, there was a Unix adage: *"Bourne to Program, Type with Joy"* FWIW: tcsh supports TOPS-20 autocomplete -- a little work with your search engine, you can figure out how to use its many options. That said, the GNU bash is said to do it also, but I can not say I have tried it personally since the ROMS in my fingers were long ago burned to 'Type with Joy.' Also in 50 years, it's so much that UNIX is perfect, it has lots of flaws and quirks. Thinking about them and considering 'better' solutions is often wise, particularly when capabilities (like Moore's law) give you new tools to solve them. But a level of wisdom here is not all of those quirks are worth repairing. In the case of command-line parsing, getopt(3) has proven to be 'good enough' for most things. If it was really as bad as you seem to think, I suspect one of the previous N attempts over the last 50 years might have taken root. My point in my previous message was that getopt(3) was created to solve the original UNIX problem. It did actually take root (I'll not get into if the Gnu long stuff was an improvement). But there were other attempts, including the Tops-20 scheme (which has been pointed out is quite similar to yours) that have been around for at least 35 years in the UNIX community and it did not catch on. I ask you to think about if maybe your value of that feature might be more than others have set it to be. As an analog, when I first came to UNIX and C from other systems, ideas like the open curly brace/close curly brace instead of BEGIN/END in C, and there were plenty of things in Ken's original shell that I found annoying, particularly coming from the regularity of TOPS-20 and the like. Hey, I used EMACS, TECO and DDT and none of them were in my new kit. But I forced myself to learn the new tools and new way of doing things. Since I was programming on UNIX in C, I made sure my code looked like everyone else [K&R did not yet exist -- but we would later call this 'White Book C." Why? So someone else could read it. I learned that style too and frankly have a hard time with any C code that does not follow it today. But if I am writing in a BEGIN/END style language, I adopt that style. When in Rome and all that. In time, the wonderful things I could do in the UNIX world way outpaced what I could do in the old world. In fact, by the time either TECO or EMACS bacame available for my use by then on a Vax, I never switched off the earlier UNIX tools I had learned. Like I said, I 'Type with Joy", frankly even if I'm on a Mac, Linux or Windows -- I switch the shell to be tcsh. Could I learn a new shell, sure? If I were to switch today, it would probably be zsh, but my suggestion is to learn the tools that system has really well. They keep using them. Adapt to the style of the system you are using. Anyway, that my thoughts from an old guy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rich.salz at gmail.com Sun Aug 1 07:06:11 2021 From: rich.salz at gmail.com (Richard Salz) Date: Sat, 31 Jul 2021 17:06:11 -0400 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <202107311920.16VJK2jT2362871@fourwinds.com> References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> Message-ID: On Sat, Jul 31, 2021 at 3:21 PM Jon Steinhart wrote: > opinion, it doesn't add value to do something that's already been done > but differently; it detracts from value because now there's yet another > competing way to do something. > You mean like not using getopt and rolling your own? Shrug. while ((i = getopt(argc, argv, "xxxxx:xxxx")) != -1) switch (i) { case .... } argc -= optind; argv += optind; So I never got getopt(). One of my rules is that I don't use a library > in cases where the number of lines of gunk that that it takes to use a > library function is >= the number of lines to just write it myself. I don't know, what lines in the above are extra beyond what you write? The last two if being generous I suppose. -------------- next part -------------- An HTML attachment was scrubbed... URL: From msi at malbolge.net Sun Aug 1 07:30:22 2021 From: msi at malbolge.net (Michael Siegel) Date: Sat, 31 Jul 2021 23:30:22 +0200 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <20210731205609.07e8149f@moon> Message-ID: <20210731233022.0960b76d@moon> Am Sat, 31 Jul 2021 15:41:17 -0400 schrieb Clem Cole : > On Sat, Jul 31, 2021 at 2:58 PM Michael Siegel > wrote: > > > I mean, which shell would I use to accomplish this on Unix? > > In the old days, when the first Unix shell wars started, there was a > Unix adage: *"Bourne to Program, Type with Joy"* > FWIW: tcsh supports TOPS-20 autocomplete -- a little work with your > search engine, you can figure out how to use its many options. That > said, the GNU bash is said to do it also, but I can not say I have > tried it personally since the ROMS in my fingers were long ago burned > to 'Type with Joy.' I see. I currently use Bash as my shell most of the time, and I have my doubts about that being a good idea. But I also doubt I would like tcsh any more. I've had a bit of experience with it on FreeBSD once. All I can say is: We didn't get along when we first met, and we haven't met since. The one and only shell I know that is (arguably) both a traditional Unix shell and a huge improvement on the traditional Unix shell is rc, which I have recently begun to use on and off. I can see myself switching to that eventually, even though it lacks some features I've come to depend on. It's definitely non-standard. But I don't care about that very much because I believe it's objectively better, and considerably so. > Also in 50 years, it's so much that UNIX is perfect, it has lots of > flaws and quirks. Thinking about them and considering 'better' > solutions is often wise, particularly when capabilities (like Moore's > law) give you new tools to solve them. But a level of wisdom here is > not all of those quirks are worth repairing. In the case of > command-line parsing, getopt(3) has proven to be 'good enough' for > most things. If it was really as bad as you seem to think, I suspect > one of the previous N attempts over the last 50 years might have > taken root. > > My point in my previous message was that getopt(3) was created to > solve the original UNIX problem. It did actually take root (I'll not > get into if the Gnu long stuff was an improvement). But there were > other attempts, including the Tops-20 scheme (which has been pointed > out is quite similar to yours) that have been around for at least 35 > years in the UNIX community and it did not catch on. I ask you to > think about if maybe your value of that feature might be more than > others have set it to be. To me, using getopt/getopts has always felt more like a way to complicate parsing rather than solving any actual problem. My aim is to get around writing an actual parsing routine based on a half-backed set of rules each time I put together a command-line utility because that is time-consuming (for no good reason) and error-prone. I really find the TOPS-20 way of going about this inspiring, though I'd aim for something way more primitive that should indeed be good enough. And I'd want it to stay as close to the POSIX Utility Syntax Guidelines as reasonably possible because even though these are lacking, I find them a reasonable base to build upon. Also, experience tells me that merely adapting to what has taken root is quite often not a good idea at all. In fact, the reasons for something good and valuable not taking root might actually turn out to be pretty nasty. > As an analog, when I first came to UNIX and C from other systems, > ideas like the open curly brace/close curly brace instead of > BEGIN/END in C, and there were plenty of things in Ken's original > shell that I found annoying, particularly coming from the regularity > of TOPS-20 and the like. Hey, I used EMACS, TECO and DDT and none of > them were in my new kit. But I forced myself to learn the new tools > and new way of doing things. Since I was programming on UNIX in C, I > made sure my code looked like everyone else [K&R did not yet exist -- > but we would later call this 'White Book C." Why? So someone else > could read it. I learned that style too and frankly have a hard > time with any C code that does not follow it today. But if I am > writing in a BEGIN/END style language, I adopt that style. When in > Rome and all that. > > In time, the wonderful things I could do in the UNIX world way > outpaced what I could do in the old world. In fact, by the time > either TECO or EMACS bacame available for my use by then on a Vax, I > never switched off the earlier UNIX tools I had learned. Like I > said, I 'Type with Joy", frankly even if I'm on a Mac, Linux or > Windows -- I switch the shell to be tcsh. Could I learn a new shell, > sure? If I were to switch today, it would probably be zsh, but my > suggestion is to learn the tools that system has really well. They > keep using them. Adapt to the style of the system you are using. As you'll be able to guess by now, I beg to differ. For example, I have forced myself to learn POSIX shell and Bash, even enjoying some of it along the way. Today, I believe that they are both rather terrible things I don't want to spend too much time with. (That said, for my use case, Bash is almost always preferable over the available POSIX sh implementation.) Then, I have always had a strong dislike for the interface of the Unix `find` command. So, I tried to replace it with what I thought was a better solution (relatively). That required me to understand `find` on a whole different level. And after gaining a much better understanding of `find` (and losing some of my dislike for it), I still believe it should be replaced and have a few ideas on how to do that. (Sadly, I mainly just have ideas.) So, in a nutshell: I think that adapting to something that you believe to be more than slightly deficient after giving it a try and trying to understand its logic is not a reasonable thing to do. > Anyway, that my thoughts from an old guy. They're much appreciated. -- Michael From jon at fourwinds.com Sun Aug 1 07:32:06 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Sat, 31 Jul 2021 14:32:06 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> Message-ID: <202107312132.16VLW6VD2410038@fourwinds.com> Richard Salz writes: > On Sat, Jul 31, 2021 at 3:21 PM Jon Steinhart wrote: > > > opinion, it doesn't add value to do something that's already been done > > but differently; it detracts from value because now there's yet another > > competing way to do something. > > > > You mean like not using getopt and rolling your own? Shrug. > > while ((i = getopt(argc, argv, "xxxxx:xxxx")) != -1) > switch (i) { > case .... > } > argc -= optind; > argv += optind; > > So I never got getopt(). One of my rules is that I don't use a library > > in cases where the number of lines of gunk that that it takes to use a > > library function is >= the number of lines to just write it myself. > > > I don't know, what lines in the above are extra beyond what you write? The > last two if being generous I suppose. Well, in my opinion that's not really an accurate representation of using getopt. I would of course write the #include line, and the table of options, which would end up being >= the number of lines that it takes me to do this... while (--argc > 0) { if (*(++argv)[0] == '-') { for (p = *argv + 1; *p != '\0'; p++) { switch (*p) { Even if it took a few more lines to do it my way, I'm a believer that good coding style keeps "meatspace locality of reference" in mind. As programmers, we put in a lot of effort to ensure locality of reference for computers, but then completely toss it for people who aren't as good as it. So given a choice of a few lines of code versus having to look something up somewhere else, I choose the few lines of code. Being a geezer, I have lots of code lying around from which I can extract working fragments such as the one above. Writing those few lines of code provides insulation from supply-side attack vectors bugs in libraries, versioning issues, having to load debug libraries, and so on. I realize that this isn't a huge deal by itself; it's a philosophical point. When I strace any random program that I didn't write I'm astonished by the amount of library loading that takes place. So any issues are multiplied by n. Don't get me wrong; I use plenty of libraries. But I tend to use those for stuff that is so common that there is a benefit from shared libraries (or at least there was before everything got containerized) and for libraries that do actual hard stuff. But I don't use libraries for small snippets of code that I could easily write myself yielding better code clarity for others reading my code. Jon From rich.salz at gmail.com Sun Aug 1 07:37:58 2021 From: rich.salz at gmail.com (Richard Salz) Date: Sat, 31 Jul 2021 17:37:58 -0400 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <202107312132.16VLW6VD2410038@fourwinds.com> References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <202107312132.16VLW6VD2410038@fourwinds.com> Message-ID: On Sat, Jul 31, 2021 at 5:34 PM Jon Steinhart wrote: > Well, in my opinion that's not really an accurate representation of using > getopt. > > It's how all my getopt code works. getopt is in libc and a stdlib.h so you can't count that against it :) on the other hand, your sample code didn't show arg/no-arg handling. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon at fourwinds.com Sun Aug 1 07:55:03 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Sat, 31 Jul 2021 14:55:03 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <202107312132.16VLW6VD2410038@fourwinds.com> Message-ID: <202107312155.16VLt3AO2429969@fourwinds.com> Richard Salz writes: > > On Sat, Jul 31, 2021 at 5:34 PM Jon Steinhart wrote: > > > Well, in my opinion that's not really an accurate representation of using > > getopt. > > > > > It's how all my getopt code works. > > getopt is in libc and a stdlib.h so you can't count that against it :) on > the other hand, your sample code didn't show arg/no-arg handling. Well, at least on my system it's here: SYNOPSIS #include not either of those other places. I could provide you with a complete working example, but I don't think that it's the important part of the discussion. Using getopt() is more or less a wash in terms of lines of code so the meatspace locality of reference argument carries the day for me. Your mileage may vary. Jon From bakul at iitbombay.org Sun Aug 1 08:04:48 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Sat, 31 Jul 2021 15:04:48 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <202107311920.16VJK2jT2362871@fourwinds.com> References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> Message-ID: <3598EE71-E652-43EF-B1AB-91CB8B3D577A@iitbombay.org> On Jul 31, 2021, at 12:20 PM, Jon Steinhart wrote: > > So I never got getopt(). One of my rules is that I don't use a library > in cases where the number of lines of gunk that that it takes to use a > library function is >= the number of lines to just write it myself. Yeah, > I know the "but the library has more eyeballs and is debugged" argument > but in reality libraries are the source of many bugs. I've always taken > the approach that I would never hire someone who had to use a library to > implement a singly-linked list. getopt() is perhaps the wrong solution but consider something like MH, whose commands all follow a common pattern. Consider: - options (switches) all start with a single '-' - they may be abbreviated to a unique prefix. - Boolean options may be inverted by prepending -no (e.g. -nolist) - value options may also have -no format to remove a previous (or default) value - options may appear anywhere and the last instance wins But different commands take different options. It would make sense to factor out common parsing, help etc. for a consistent treatment. In my Go code for parsing MH like options I used Go's flag package as a model. Personally I vastly prefer MH style option processing to either single char options or --very-long-names which can't be abbreviated. Never mind options for commands like gcc, who can even remember 40+ ls options? But I haven't thought about how to extend this for shell scripts, or exposing these so that shells like zsh can do command completion. To specify these you need a vector of tuples (name, type, default, brief-help) but that is painful to do in a shell. From imp at bsdimp.com Sun Aug 1 08:10:04 2021 From: imp at bsdimp.com (Warner Losh) Date: Sat, 31 Jul 2021 16:10:04 -0600 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <202107312132.16VLW6VD2410038@fourwinds.com> References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <202107312132.16VLW6VD2410038@fourwinds.com> Message-ID: On Sat, Jul 31, 2021 at 3:33 PM Jon Steinhart wrote: > Richard Salz writes: > > On Sat, Jul 31, 2021 at 3:21 PM Jon Steinhart wrote: > > > > > opinion, it doesn't add value to do something that's already been done > > > but differently; it detracts from value because now there's yet another > > > competing way to do something. > > > > > > > You mean like not using getopt and rolling your own? Shrug. > > > > while ((i = getopt(argc, argv, "xxxxx:xxxx")) != -1) > > switch (i) { > > case .... > > } > > argc -= optind; > > argv += optind; > > > > So I never got getopt(). One of my rules is that I don't use a library > > > in cases where the number of lines of gunk that that it takes to use a > > > library function is >= the number of lines to just write it myself. > > > > > > I don't know, what lines in the above are extra beyond what you write? > The > > last two if being generous I suppose. > > Well, in my opinion that's not really an accurate representation of using > getopt. > > I would of course write the #include line, and the table of options, which > would > end up being >= the number of lines that it takes me to do this... > > while (--argc > 0) { > if (*(++argv)[0] == '-') { > for (p = *argv + 1; *p != '\0'; p++) { > switch (*p) { > Except for all the things this gets wrong, it's ok. The problem with inlining getopt is that you wind up with cases like -f foo'' on the command line being treated differently than '-ffoo'. Inlined code like this can be quite frustrating for the user to use. Your locality of reference is cut and paste bugs that getopt eliminates because it handles all the special cases in a uniform way. > Even if it took a few more lines to do it my way, I'm a believer that good > coding > style keeps "meatspace locality of reference" in mind. As programmers, we > put in > a lot of effort to ensure locality of reference for computers, but then > completely > toss it for people who aren't as good as it. So given a choice of a few > lines of > code versus having to look something up somewhere else, I choose the few > lines of > code. > And a few more bugs... Being a geezer, I have lots of code lying around from which I can extract > working > fragments such as the one above. Writing those few lines of code provides > insulation > from supply-side attack vectors bugs in libraries, versioning issues, > having to load > debug libraries, and so on. > getopt has been standardized since the 80s and has had universal adoption since the 90s. Hardly a version chasing issue since it's in everybody's libc. > I realize that this isn't a huge deal by itself; it's a philosophical > point. When > I strace any random program that I didn't write I'm astonished by the > amount of > library loading that takes place. So any issues are multiplied by n. > The flip side to this is that libraries can be debugged once, while inline code like the above needs to be deugged over and over.... > Don't get me wrong; I use plenty of libraries. But I tend to use those > for stuff > that is so common that there is a benefit from shared libraries (or at > least there > was before everything got containerized) and for libraries that do actual > hard stuff. > But I don't use libraries for small snippets of code that I could easily > write > myself yielding better code clarity for others reading my code. > Given the number of times I've been burned by trying to roll my own getopt, I stopped trying years ago. It's harder than it looks. Warner > Jon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lm at mcvoy.com Sun Aug 1 08:13:17 2021 From: lm at mcvoy.com (Larry McVoy) Date: Sat, 31 Jul 2021 15:13:17 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <3598EE71-E652-43EF-B1AB-91CB8B3D577A@iitbombay.org> References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <3598EE71-E652-43EF-B1AB-91CB8B3D577A@iitbombay.org> Message-ID: <20210731221317.GG23759@mcvoy.com> On Sat, Jul 31, 2021 at 03:04:48PM -0700, Bakul Shah wrote: > On Jul 31, 2021, at 12:20 PM, Jon Steinhart wrote: > > > > So I never got getopt(). One of my rules is that I don't use a library > > in cases where the number of lines of gunk that that it takes to use a > > library function is >= the number of lines to just write it myself. Yeah, > > I know the "but the library has more eyeballs and is debugged" argument > > but in reality libraries are the source of many bugs. I've always taken > > the approach that I would never hire someone who had to use a library to > > implement a singly-linked list. > > getopt() is perhaps the wrong solution but consider something like MH, > whose commands all follow a common pattern. Consider: > > - options (switches) all start with a single '-' > - they may be abbreviated to a unique prefix. That last one is a gotcha waiting to happen: program --this-is-the-long-option is the same as program --this but that will break scripts (and fingers) when program gets a new option like program --this-is-the-even-longer-option We wrote our own getopt() for BitKeeper and it had long and short options but no gotcha unique prefix. From bakul at iitbombay.org Sun Aug 1 08:14:41 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Sat, 31 Jul 2021 15:14:41 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <20210731221317.GG23759@mcvoy.com> References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <3598EE71-E652-43EF-B1AB-91CB8B3D577A@iitbombay.org> <20210731221317.GG23759@mcvoy.com> Message-ID: On Jul 31, 2021, at 3:13 PM, Larry McVoy wrote: > > On Sat, Jul 31, 2021 at 03:04:48PM -0700, Bakul Shah wrote: >> On Jul 31, 2021, at 12:20 PM, Jon Steinhart wrote: >>> >>> So I never got getopt(). One of my rules is that I don't use a library >>> in cases where the number of lines of gunk that that it takes to use a >>> library function is >= the number of lines to just write it myself. Yeah, >>> I know the "but the library has more eyeballs and is debugged" argument >>> but in reality libraries are the source of many bugs. I've always taken >>> the approach that I would never hire someone who had to use a library to >>> implement a singly-linked list. >> >> getopt() is perhaps the wrong solution but consider something like MH, >> whose commands all follow a common pattern. Consider: >> >> - options (switches) all start with a single '-' >> - they may be abbreviated to a unique prefix. > > That last one is a gotcha waiting to happen: > > program --this-is-the-long-option > > is the same as > > program --this > > but that will break scripts (and fingers) when program gets a new > option like That is easy to fix: use full options in scripts. Abbreviations for interactive use. Much better than --always-having-to-type-long-names. > > program --this-is-the-even-longer-option > > We wrote our own getopt() for BitKeeper and it had long and short options > but no gotcha unique prefix. From jon at fourwinds.com Sun Aug 1 08:16:58 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Sat, 31 Jul 2021 15:16:58 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <3598EE71-E652-43EF-B1AB-91CB8B3D577A@iitbombay.org> References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <3598EE71-E652-43EF-B1AB-91CB8B3D577A@iitbombay.org> Message-ID: <202107312216.16VMGwTX2446561@fourwinds.com> Bakul Shah writes: > On Jul 31, 2021, at 12:20 PM, Jon Steinhart wrote: > > > > So I never got getopt(). One of my rules is that I don't use a library > > in cases where the number of lines of gunk that that it takes to use a > > library function is >= the number of lines to just write it myself. Yeah, > > I know the "but the library has more eyeballs and is debugged" argument > > but in reality libraries are the source of many bugs. I've always taken > > the approach that I would never hire someone who had to use a library to > > implement a singly-linked list. > > getopt() is perhaps the wrong solution but consider something like MH, > whose commands all follow a common pattern. Consider: > > - options (switches) all start with a single '-' > - they may be abbreviated to a unique prefix. > - Boolean options may be inverted by prepending -no (e.g. -nolist) > - value options may also have -no format to remove a previous (or default) value > - options may appear anywhere and the last instance wins > > But different commands take different options. It would make sense to factor > out common parsing, help etc. for a consistent treatment. In my Go code for > parsing MH like options I used Go's flag package as a model. > > Personally I vastly prefer MH style option processing to either single char > options or --very-long-names which can't be abbreviated. Never mind options for > commands like gcc, who can even remember 40+ ls options? > > But I haven't thought about how to extend this for shell scripts, or > exposing these so that shells like zsh can do command completion. To specify > these you need a vector of tuples (name, type, default, brief-help) but that > is painful to do in a shell. Ah, well, you've given away the secret of real UNIX geezers, we're on both this mailing list and the nmh list :-) Yes, I'm mostly happy with the way that nmh does options. I guess that I would look more kindly on getopt if it had existed much earlier so that people writing new commands would be encouraged to use the same format. Not as happy with it as an afterthought. Once again, I have to go back to the meatspace locality of reference issues. Sure, it would be nice to be able to factor out common parsing, for example if a related set of programs shared the same option set. But unless it's something huge, I'd just put it in it's own file and use it for multiple programs; I wouldn't put it in a library. My point is that the code that does the actual parsing is really trivial, and not necessarily the best use of a library. As far as help goes, I don't expect help built into command line programs; I expect to look up error messages on the manual pages. I'm happy with a generic usage error as most "helpful" output that I get from programs is not actually helpful. From bakul at iitbombay.org Sun Aug 1 08:17:13 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Sat, 31 Jul 2021 15:17:13 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <3598EE71-E652-43EF-B1AB-91CB8B3D577A@iitbombay.org> <20210731221317.GG23759@mcvoy.com> Message-ID: <68F654B3-7192-4E95-B09E-3B421A11E328@iitbombay.org> On Jul 31, 2021, at 3:14 PM, Bakul Shah wrote: > > On Jul 31, 2021, at 3:13 PM, Larry McVoy wrote: >> >> On Sat, Jul 31, 2021 at 03:04:48PM -0700, Bakul Shah wrote: >>> On Jul 31, 2021, at 12:20 PM, Jon Steinhart wrote: >>>> >>>> So I never got getopt(). One of my rules is that I don't use a library >>>> in cases where the number of lines of gunk that that it takes to use a >>>> library function is >= the number of lines to just write it myself. Yeah, >>>> I know the "but the library has more eyeballs and is debugged" argument >>>> but in reality libraries are the source of many bugs. I've always taken >>>> the approach that I would never hire someone who had to use a library to >>>> implement a singly-linked list. >>> >>> getopt() is perhaps the wrong solution but consider something like MH, >>> whose commands all follow a common pattern. Consider: >>> >>> - options (switches) all start with a single '-' >>> - they may be abbreviated to a unique prefix. >> >> That last one is a gotcha waiting to happen: >> >> program --this-is-the-long-option >> >> is the same as >> >> program --this >> >> but that will break scripts (and fingers) when program gets a new >> option like > > That is easy to fix: use full options in scripts. Abbreviations for > interactive use. Much better than --always-having-to-type-long-names. > >> >> program --this-is-the-even-longer-option >> >> We wrote our own getopt() for BitKeeper and it had long and short options >> but no gotcha unique prefix. Forgot to add that whoever extends "program" should know not to create a new option that uses a longer name breaking a full form old options. From lm at mcvoy.com Sun Aug 1 08:19:50 2021 From: lm at mcvoy.com (Larry McVoy) Date: Sat, 31 Jul 2021 15:19:50 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <202107312132.16VLW6VD2410038@fourwinds.com> Message-ID: <20210731221950.GH23759@mcvoy.com> On Sat, Jul 31, 2021 at 04:10:04PM -0600, Warner Losh wrote: > On Sat, Jul 31, 2021 at 3:33 PM Jon Steinhart wrote: > > > Richard Salz writes: > > > On Sat, Jul 31, 2021 at 3:21 PM Jon Steinhart wrote: > > > > > > > opinion, it doesn't add value to do something that's already been done > > > > but differently; it detracts from value because now there's yet another > > > > competing way to do something. > > > > > > > > > > You mean like not using getopt and rolling your own? Shrug. > > > > > > while ((i = getopt(argc, argv, "xxxxx:xxxx")) != -1) > > > switch (i) { > > > case .... > > > } > > > argc -= optind; > > > argv += optind; > > > > > > So I never got getopt(). One of my rules is that I don't use a library > > > > in cases where the number of lines of gunk that that it takes to use a > > > > library function is >= the number of lines to just write it myself. > > > > > > > > > I don't know, what lines in the above are extra beyond what you write? > > The > > > last two if being generous I suppose. > > > > Well, in my opinion that's not really an accurate representation of using > > getopt. > > > > I would of course write the #include line, and the table of options, which > > would > > end up being >= the number of lines that it takes me to do this... > > > > while (--argc > 0) { > > if (*(++argv)[0] == '-') { > > for (p = *argv + 1; *p != '\0'; p++) { > > switch (*p) { > > > > Except for all the things this gets wrong, it's ok. The problem with > inlining getopt > is that you wind up with cases like -f foo'' on the command line being > treated differently > than '-ffoo'. BitKeeper's getopt had a different char for that: "f:" allows -ffoo or -f foo but "f;" insists on no space. With that, I'm bowing out of this thread, it's becoming a bike shed. From bakul at iitbombay.org Sun Aug 1 08:20:15 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Sat, 31 Jul 2021 15:20:15 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <202107312216.16VMGwTX2446561@fourwinds.com> References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <3598EE71-E652-43EF-B1AB-91CB8B3D577A@iitbombay.org> <202107312216.16VMGwTX2446561@fourwinds.com> Message-ID: -- Bakul > On Jul 31, 2021, at 3:16 PM, Jon Steinhart wrote: > > Bakul Shah writes: >> On Jul 31, 2021, at 12:20 PM, Jon Steinhart wrote: >>> >>> So I never got getopt(). One of my rules is that I don't use a library >>> in cases where the number of lines of gunk that that it takes to use a >>> library function is >= the number of lines to just write it myself. Yeah, >>> I know the "but the library has more eyeballs and is debugged" argument >>> but in reality libraries are the source of many bugs. I've always taken >>> the approach that I would never hire someone who had to use a library to >>> implement a singly-linked list. >> >> getopt() is perhaps the wrong solution but consider something like MH, >> whose commands all follow a common pattern. Consider: >> >> - options (switches) all start with a single '-' >> - they may be abbreviated to a unique prefix. >> - Boolean options may be inverted by prepending -no (e.g. -nolist) >> - value options may also have -no format to remove a previous (or default) value >> - options may appear anywhere and the last instance wins >> >> But different commands take different options. It would make sense to factor >> out common parsing, help etc. for a consistent treatment. In my Go code for >> parsing MH like options I used Go's flag package as a model. >> >> Personally I vastly prefer MH style option processing to either single char >> options or --very-long-names which can't be abbreviated. Never mind options for >> commands like gcc, who can even remember 40+ ls options? >> >> But I haven't thought about how to extend this for shell scripts, or >> exposing these so that shells like zsh can do command completion. To specify >> these you need a vector of tuples (name, type, default, brief-help) but that >> is painful to do in a shell. > > Ah, well, you've given away the secret of real UNIX geezers, we're on both > this mailing list and the nmh list :-) :-) > Yes, I'm mostly happy with the way that nmh does options. > > I guess that I would look more kindly on getopt if it had existed much earlier > so that people writing new commands would be encouraged to use the same format. > Not as happy with it as an afterthought. > > Once again, I have to go back to the meatspace locality of reference issues. > Sure, it would be nice to be able to factor out common parsing, for example > if a related set of programs shared the same option set. But unless it's > something huge, I'd just put it in it's own file and use it for multiple > programs; I wouldn't put it in a library. My point is that the code that > does the actual parsing is really trivial, and not necessarily the best > use of a library. > > As far as help goes, I don't expect help built into command line programs; > I expect to look up error messages on the manual pages. I'm happy with a > generic usage error as most "helpful" output that I get from programs is > not actually helpful. Note that -help in MH program is far more useful as it spells out the full option name. Consider % refile -he Usage: refile [msgs] [switches] +folder ... switches are: -draft -[no]link -[no]preserve -[no]retainsequences -[no]unlink -src +folder -file file -rmmproc program -normmproc -version -help ... vs % ls -z ls: invalid option -- z usage: ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwxy1,] [--color=when] [-D format] [file ...] From jon at fourwinds.com Sun Aug 1 08:20:13 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Sat, 31 Jul 2021 15:20:13 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <202107312132.16VLW6VD2410038@fourwinds.com> Message-ID: <202107312220.16VMKDd92449332@fourwinds.com> Warner Losh writes: > > The flip side to this is that libraries can be debugged once, while inline > code like the above needs to be deugged over and over.... Well, no. Inline code doesn't need to be debugged over and over. It doesn't have to be written from scratch every time. While in theory your point about libraries is correct, it hasn't seem to have worked out in practice. Better in C than in node.js, but there have been plenty of spectacular bugs found in old C libraries recently. Jon From imp at bsdimp.com Sun Aug 1 09:26:00 2021 From: imp at bsdimp.com (Warner Losh) Date: Sat, 31 Jul 2021 17:26:00 -0600 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <202107312220.16VMKDd92449332@fourwinds.com> References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <202107312132.16VLW6VD2410038@fourwinds.com> <202107312220.16VMKDd92449332@fourwinds.com> Message-ID: On Sat, Jul 31, 2021 at 4:37 PM Jon Steinhart wrote: > Warner Losh writes: > > > > The flip side to this is that libraries can be debugged once, while > inline > > code like the above needs to be deugged over and over.... > > Well, no. Inline code doesn't need to be debugged over and over. It > doesn't > have to be written from scratch every time. While in theory your point > about > libraries is correct, it hasn't seem to have worked out in practice. > Better > in C than in node.js, but there have been plenty of spectacular bugs found > in > old C libraries recently. > The large number of times I've had to replace inline code like you've quoted with getopt to fix the irregularities in command line parsing suggests that we differ on this viewpoint. Warner -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon at fourwinds.com Sun Aug 1 09:41:15 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Sat, 31 Jul 2021 16:41:15 -0700 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: References: <20210731142533.69caf929@moon> <202107311920.16VJK2jT2362871@fourwinds.com> <202107312132.16VLW6VD2410038@fourwinds.com> <202107312220.16VMKDd92449332@fourwinds.com> Message-ID: <202107312341.16VNfFRb2521415@fourwinds.com> Warner Losh writes: > The large number of times I've had to replace inline code like you've > quoted with > getopt to fix the irregularities in command line parsing suggests that we > differ on > this viewpoint. Fine by me. Never hurts to know what other people consider best practices. From msi at malbolge.net Mon Aug 2 02:51:37 2021 From: msi at malbolge.net (Michael Siegel) Date: Sun, 1 Aug 2021 18:51:37 +0200 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: <20210801185137.0a4822b1@moon> Am Sat, 31 Jul 2021 11:56:51 -0400 schrieb Paul Winalski : > On 7/31/21, Michael Siegel wrote: > > > > While doing that, I learned that there is a better way to approach > > this problem – beyond using getopt(s) (which never really made > > sense to me) and having to write case statements in loops every > > time: Define a grammar, let a pre-built parser do the work, and > > have the parser provide the results to the program. > > This method for handling command lines dates back at least to the > 1970s. The COMND JSYS (system call) in TOPS-20 operated this way, as > does the DCL command line interface in OpenVMS. As you pointed out it > can greatly simplify the code in the application. It also permits > command completion. If the command has a long-winded option, such as > -supercalifragilisticexpialidocious, I can type -super then hit the > TAB key and as long as there is only one option that starts with > -super the parser will fill in the rest of the long keyword. It also > means that you can provide interactive help. At any point the user > can type a question mark and the command interpreter will say what > syntactic element is expected next. Being able to provide interactive help is exactly what the person who suggested grammar-based parsing to me was working on. I hadn't even thought about that at first. But given my recent investigation into built-in command documentation on Unix-like systems, I tend to think this would be a great enhancement – if it was implemented with a strict focus on not getting in the way, i.e., the user should be able to switch it off completely, and search-as-you-type would be opt-in, if provided at all. -- Michael From jon at fourwinds.com Mon Aug 2 03:31:31 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Sun, 01 Aug 2021 10:31:31 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <20210801185137.0a4822b1@moon> References: <20210731142533.69caf929@moon> <20210801185137.0a4822b1@moon> Message-ID: <202108011731.171HVWPG3452003@fourwinds.com> Michael Siegel writes: > Being able to provide interactive help is exactly what the person who > suggested grammar-based parsing to me was working on. I hadn't even > thought about that at first. But given my recent investigation into > built-in command documentation on Unix-like systems, I tend to think > this would be a great enhancement – if it was implemented with a > strict focus on not getting in the way, i.e., the user should be able > to switch it off completely, and search-as-you-type would be opt-in, if > provided at all. > > > -- > Michael While I agree with you in theory, I'm dubious about how it would work in practice. Sorry if it sounds like I've lost faith with my profession but I'm trying to accept reality. Where would such documentation come from? Not wanting to reopen old flame wars, but the fragmentation of the documentation system where some things are man pages, some are info pages, some are random HTML files, some are only online, some things having no documentation at all except maybe a help message, and much of that have no actual content, is reality. Who's going to write more documentation, and how is it going to be kept consistent with other documentation? Is your help system going to be yet another fragment? Also, we seem to be well into a "Farenheight 451" world now where a vast number of people communicate only by photos and video. Writing seems to be deprecated. I don't have confidence that any documentation would be useful even if people wrote it. As a current example, I'm having trouble with a btrfs filesystem and I can't say that the btrfs-check manual page contains any useful content. Maybe since Microsoft "AI" is now going to write code for, not sure what to call them, programmers doesn't seem right any more, maybe it'll write their documentation too? I guess what I'm saying is that it sounds like you are having some good thoughts on a technical solution that I think will fail without also having a social solution. If you could somehow extract your help info from man pages without creating a whole new documentation system it might work in the few cases where there are good manual pages. Wow I sound grumpy this morning. Jon From chet.ramey at case.edu Mon Aug 2 03:44:12 2021 From: chet.ramey at case.edu (Chet Ramey) Date: Sun, 1 Aug 2021 13:44:12 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> Message-ID: <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> On 7/31/21 12:19 PM, Dan Cross wrote: > There was someone posting here on TUHS a while back about leveraging a > special context-sensitive `--shell-help` or similar command line program > and synthesizing a protocol between the shell and a program to provide > TOPS-20 like command completion. It was nowhere near what you get from the > COMND JSYS, but seemed like a reasonable approximation. This is essentially how the existing shells do it (bash, zsh, tcsh, etc.), but in an ad-hoc fashion. There is no standard way to obtain possible completions or list possible arguments, so the shells push that to external generators. Since you have to perform the completion in the shell, there has to be some way to tell the shell the possible completions for each command of interest, whether that's options or arguments. The different shells have solved that in essentially the same way, with a few syntactic variations. Bash provides a framework (complete/compgen/compctl) and pushes a lot of the command-specific work to external completers. It provides access to the shell internals (lists of builtins, functions, aliases, variables, and so on) and built-in ways to perform common completions (filenames, directory names, command names, etc.), and leaves the rest to external commands or shell functions. The real power and flexibility comes from being able to invoke these external commands or shell functions to generate lists of possible completions, and defining an API between the shell and those generators to specify enough of the command line to make it easy to find the word to be completed, the command for which completion is being attempted, and clarifying context around that word. In the same way, the shell provides an API for those generators to return possible completions. The knowledge about each command's options and arguments is embedded in these generators. A standard way to handle command line options and arguments would make generators easier to write, but doesn't address the other issues of what, exactly, the user wants to complete, so the existing mechanisms would likely not change very much. Something like `--shell-help', as long as it were context-sensitive, would help more. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet at case.edu http://tiswww.cwru.edu/~chet/ From chet.ramey at case.edu Mon Aug 2 03:48:19 2021 From: chet.ramey at case.edu (Chet Ramey) Date: Sun, 1 Aug 2021 13:48:19 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <20210731205609.07e8149f@moon> References: <20210731142533.69caf929@moon> <20210731205609.07e8149f@moon> Message-ID: <656b68de-fc95-d75f-1d93-8d6410296d00@case.edu> On 7/31/21 2:56 PM, Michael Siegel wrote: > Am Sat, 31 Jul 2021 10:30:18 -0700 > schrieb Anthony Martin : > >> Michael Siegel once said: >>> So, I've prepared a bit of a write-up, pondering on the pros and >>> cons of two different ways of having task-specific tool sets >>> (non-hierarchical command sets vs. sub-commands) that is available >>> at >>> >>> https://www.msiism.org/files/doc/unix-like_command-line_interfaces.html >>> >>> I tend to think the sub-command approach is better. But I'm neither >>> a UI nor a Unix expert and have no formal training in computer >>> things. So, I thought this would be a good place to ask for comment >>> (and get some historical perspective). >> >> You're missing the approach taken in Plan 9 (and >> 10th edition Unix): put related commands in a >> directory and use a shell that doesn't restrict >> the first argument of a command to a single path >> element. >> >> This lets you execute commands like: >> >> auth/as >> disk/prep >> git/rebase >> ip/ping >> ndb/dns >> upas/send >> >> without having a prefix on every command name or >> single large binaries with every command linked >> in as subcommands. > > Thanks for pointing this out. I had no idea. > > Unfortunately(?), I'm looking to make my life easier on more "Unix-like > Unix-like systems" (for want of a better term), for the time being > (Linux, BSD, maybe illumos). (I mean, which shell would I use to > accomplish this on Unix?) POSIX forbids this behavior, FWIW, so you'll probably have a hard time finding a shell -- at least one with POSIX aspirations -- that implements it. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet at case.edu http://tiswww.cwru.edu/~chet/ From douglas.mcilroy at dartmouth.edu Mon Aug 2 04:17:39 2021 From: douglas.mcilroy at dartmouth.edu (Douglas McIlroy) Date: Sun, 1 Aug 2021 14:17:39 -0400 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] Message-ID: I have considerable sympathy with the general idea of formally specifying and parsing inputs. Langsec people make a strong case for doing so. The white paper,"A systematic approach to modern Unix-like command interfaces", proposes to "simplify parsing by facilitating the creation of easy-to-use 'grammar-based' parsers". I'm not clear on what is meant by "parser". A plain parser is a beast that builds a parse tree according to a grammar. For most standard Unix programs, the parse tree has two kinds of leaves: non-options and options with k parameters. Getopt restricts k to {0,1}. Aside from fiddling with argc and argv, I see little difference in working with a parse tree for arguments that could be handled by getopt and working with using getopt directly. A more general parser could handle more elaborate grammatic constraints on options, for example, field specs in sort(1), requirements on presence of options in tar(1), or representation of multiple parameters in cut(1). In realizing the white paper's desire to "have the parser provide the results to the program", it's likely that the mechanism will, like Yacc, go beyond parsing and invoke semantic actions as it identifies tree nodes. Pioneer Yaccification of some commands might be a worthy demo. Doug From rich.salz at gmail.com Mon Aug 2 05:23:10 2021 From: rich.salz at gmail.com (Richard Salz) Date: Sun, 1 Aug 2021 15:23:10 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <656b68de-fc95-d75f-1d93-8d6410296d00@case.edu> References: <20210731142533.69caf929@moon> <20210731205609.07e8149f@moon> <656b68de-fc95-d75f-1d93-8d6410296d00@case.edu> Message-ID: > >> This lets you execute commands like: > >> > >> auth/as > ... POSIX forbids this behavior, FWIW, so you'll probably have a hard time > finding a shell -- at least one with POSIX aspirations -- that implements > it. > So you write a function or alias that prepends the full path to "as" and exec's the command. So you have to type "auth as ..." but BFD. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnold at skeeve.com Mon Aug 2 05:48:10 2021 From: arnold at skeeve.com (arnold at skeeve.com) Date: Sun, 01 Aug 2021 13:48:10 -0600 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: References: Message-ID: <202108011948.171JmAcK001895@freefriends.org> Douglas McIlroy wrote: > In realizing the white paper's desire to "have the parser > provide the results to the program", it's likely that the mechanism > will, like Yacc, go beyond parsing and invoke semantic actions > as it identifies tree nodes. I have to admit that all this feels like overkill. Parsing options is only a very small part of the real work that a program does. Speaking for myself, I want something simple and regular that will get the job done and let me get on with the actual business of my software. A grammar just for command-line argument parsing feels like the tail wagging the dog: not nearly enough ROI, at least for me. I happen to like the getopt_long interface designed by the GNU project. It's easy to learn, setup and use. Once it's in place it's set and forget. My two cents, Arnold From cowan at ccil.org Mon Aug 2 07:30:19 2021 From: cowan at ccil.org (John Cowan) Date: Sun, 1 Aug 2021 17:30:19 -0400 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <202108011948.171JmAcK001895@freefriends.org> References: <202108011948.171JmAcK001895@freefriends.org> Message-ID: On Sun, Aug 1, 2021 at 3:48 PM wrote: > I happen to like the getopt_long interface designed by the GNU > project. It's easy to learn, setup and use. Once it's in place > it's set and forget. > I agree, and what is more, I say, it is a grammar already, if a simple one. You declare what you accept and what's to be done, making it a DSL expressed as an array of structs. The only thing it lacks is that old getopt is a bag on the side rather than being integrated: struct option should have an additional member "char short_option", where '\0' means "no short option". Given that feature and three per-program values "progname" (argv[0] by default), "version", and "usage_string", the --version and --help options can be processed inside getopt itself. I especially like that you pass per-option pointers saying where to put the value, so no case statement required, just create some global or local variables and pass in their addresses. Automatic support for "--nofoo" given "--foo" would be good as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From crossd at gmail.com Mon Aug 2 07:53:59 2021 From: crossd at gmail.com (Dan Cross) Date: Sun, 1 Aug 2021 17:53:59 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On Sun, Aug 1, 2021 at 1:44 PM Chet Ramey wrote: > On 7/31/21 12:19 PM, Dan Cross wrote: > > There was someone posting here on TUHS a while back about leveraging a > > special context-sensitive `--shell-help` or similar command line program > > and synthesizing a protocol between the shell and a program to provide > > TOPS-20 like command completion. It was nowhere near what you get from > the > > COMND JSYS, but seemed like a reasonable approximation. > > This is essentially how the existing shells do it (bash, zsh, tcsh, etc.), > but in an ad-hoc fashion. There is no standard way to obtain possible > completions or list possible arguments, so the shells push that to external > generators. > > Since you have to perform the completion in the shell, there has to be some > way to tell the shell the possible completions for each command of > interest, whether that's options or arguments. The different shells have > solved that in essentially the same way, with a few syntactic variations. > > Bash provides a framework (complete/compgen/compctl) and pushes a lot of > the command-specific work to external completers. It provides access to the > shell internals (lists of builtins, functions, aliases, variables, and so > on) and built-in ways to perform common completions (filenames, directory > names, command names, etc.), and leaves the rest to external commands or > shell functions. > > The real power and flexibility comes from being able to invoke these > external commands or shell functions to generate lists of possible > completions, and defining an API between the shell and those generators to > specify enough of the command line to make it easy to find the word to be > completed, the command for which completion is being attempted, and > clarifying context around that word. In the same way, the shell provides an > API for those generators to return possible completions. > > The knowledge about each command's options and arguments is embedded in > these generators. > > A standard way to handle command line options and arguments would make > generators easier to write, but doesn't address the other issues of what, > exactly, the user wants to complete, so the existing mechanisms would > likely not change very much. Something like `--shell-help', as long as it > were context-sensitive, would help more. Thanks for the useful background information on existing solutions. If I understood the proposal correctly, it was that the program in question would, itself, be the generator as described above. Perhaps it was coupled with a standard structured format for consumption by the shell, which seems like it would be useful for this sort of expansion. Of course, the process model in TOPS-20 was very different than in Unix, and in that system, as soon as you typed the _name_ of a command it's image was "run up" in your process. So the interactive help system was provided by a running instance of the program itself. What I gathered from the proposed model was that it involved multiple invocations of the program, but with a special option that would trigger behavior informally described as, "here's the context I've built so far; let me know what options are available here." I don't know that it's terribly "Unixy", but I can see how it would be useful for interactive use. As an aside, I maintain some older "machines" at home (even modest hardware can emulate a PDP-10 or Honeywell DPS8), and find that doing so provides me with perspective that can be very useful. Looking at other systems that were available roughly around the time of Unix (TENEX, Multics), it strikes me that the Unix was a bit of an odd-duck with the way it handled exec in terms of destructively overlaying the memory of the user portion of a process with a new image; am I wrong here? I wonder why the "one program per process and exec destroys what was running before" mechanism was implemented? I can imagine it had a lot to do with the constraints that early Unix machines must have imposed on design, not to mention implementation simplicity, but I wonder what the designers thought of other systems' process models and whether they were considered at all? Perhaps Doug and Ken might have thoughts here? - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chet.ramey at case.edu Mon Aug 2 09:21:16 2021 From: chet.ramey at case.edu (Chet Ramey) Date: Sun, 1 Aug 2021 19:21:16 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: <2322d5c2-19cc-d0e9-9a62-364138b9ba58@case.edu> On 8/1/21 5:53 PM, Dan Cross wrote: > Thanks for the useful background information on existing solutions. > > If I understood the proposal correctly, it was that the program in question > would, itself, be the generator as described above. Perhaps it was coupled > with a standard structured format for consumption by the shell, which seems > like it would be useful for this sort of expansion. Yes, it would make writing generators easier. The rest of the process would change very little: determining the word to complete, determining the command name, breaking the edit line into words for the generator, invoking the generator through the appropriate mechanism, parsing the results, and processing the matches. From the shell's perspective, it's a minor change. > Of course, the process model in TOPS-20 was very different than in Unix, > and in that system, as soon as you typed the _name_ of a command it's image > was "run up" in your process. So the interactive help system was provided > by a running instance of the program itself. What I gathered from the > proposed model was that it involved multiple invocations of the program, > but with a special option that would trigger behavior informally described > as, "here's the context I've built so far; let me know what options are > available here." I don't know that it's terribly "Unixy", but I can see how > it would be useful for interactive use. Yes. None of this is very "Unixy", but people have gotten used to being able to use capabilities like completion. When you're running interactively, running additional processes when you're performing word completion isn't particularly expensive. Again from the shell's perspective, invoking one generator that executes a program with `--shell-help' isn't that much different or more expensive -- and simpler in some ways because you don't have to save any incremental parsing state -- than executing a shell function that runs several processes, mostly command substitutions. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet at case.edu http://tiswww.cwru.edu/~chet/ From chet.ramey at case.edu Mon Aug 2 09:26:08 2021 From: chet.ramey at case.edu (Chet Ramey) Date: Sun, 1 Aug 2021 19:26:08 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <20210731205609.07e8149f@moon> <656b68de-fc95-d75f-1d93-8d6410296d00@case.edu> Message-ID: On 8/1/21 3:23 PM, Richard Salz wrote: > > >> This lets you execute commands like: > >> > >>      auth/as > ... > > POSIX forbids this behavior, FWIW, so you'll probably have a hard time > finding a shell -- at least one with POSIX aspirations -- that > implements it. > > > So you write a function or alias that prepends the full path to "as" and > exec's the command.  So you have to type "auth as ..." but BFD. Sure. If you invest effort in building a solution, you can do just about anything. If you want, you can write a function that generates the set of aliases you use to do this. The thing is, you're going to have to build it -- you can't expect to find a shell that does a $PATH search for a pathname containing a slash. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet at case.edu http://tiswww.cwru.edu/~chet/ From cowan at ccil.org Mon Aug 2 09:36:53 2021 From: cowan at ccil.org (John Cowan) Date: Sun, 1 Aug 2021 19:36:53 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On Sun, Aug 1, 2021 at 5:55 PM Dan Cross wrote: > Looking at other systems that were available roughly around the time of > Unix (TENEX, Multics), it strikes me that the Unix was a bit of an odd-duck > with the way it handled exec in terms of destructively overlaying the > memory of the user portion of a process with a new image; am I wrong here? > See dmr's paper at for details, but in short exec and its equivalents elsewhere have always overlaid the running program with another program. Early versions of PDP-7 Linux used the same process model as Tenex: one process per terminal which alternated between running the shell and a user program. So exec() loaded the user program on top of the shell. Indeed, this wasn't even a syscall; the shell itself wrote a tiny program loader into the top of memory that read the new program, which was open for reading, and jumped to it. Likewise, exit() was a specialized exec() that reloaded the shell. The Tenex and Multics shells had more memory to play with and didn't have to use these self-overlaying tricks[*]: they loaded your program into available memory and called it as a subroutine, which accounts for the name "shell". So it was the introduction of fork(), which came from the Berkeley Genie OS, that made the current process control regime possible. In those days, fork() wrote the current process out to the swapping disk and set up the process table with a new entry. For efficiency, the in-memory version became the child and the swapped-out version became the parent. Instantly the shell was able to run background processes by just not waiting for them, and pipelines (once the syntax was invented) could be handled with N - 1 processes in an N-stage pipeline. Huge new powers landed on the user's head. Nowadays it's a question whether fork() makes sense any more. "A fork() in the road" [Baumann et al. 2019] < https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf> is an interesting argument against fork(): * It doesn't compose. * It is insecure by default. * It is slow (there are about 25 properties a process has in addition to its memory and hardware state, and each of these needs to be copied or not) even using COW (which is itself a Good Thing and can and should be provided separately) * It is incompatible with a single-address-space design. In short, spawn() beats fork() like a drum, and fork() should be deprecated. To be sure, the paper comes out of Microsoft Research, but I find it pretty compelling anyway. [*] My very favorite self-overlaying program was the PDP-8 bootstrap for the DF32 disk drive. You toggled in two instructions at locations 30 and 31 meaning "load disk registers and go" and "jump to self" respectively, hit the Clear key on the front panel, which cleared all registers, and started up at 30. The first instruction told the disk to start reading sector 0 of the disk into location 0 in memory (because all the registers were 0, including the disk instruction register where 0 = READ) and the second instruction kept the CPU busy waiting. As the sector loaded, the two instructions were overwritten by "skip if disk ready" and "jump to previous address", which would wait until the whole sector was loaded. Then the OS could be loaded using the primitive disk driver in block 0. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lm at mcvoy.com Mon Aug 2 09:49:50 2021 From: lm at mcvoy.com (Larry McVoy) Date: Sun, 1 Aug 2021 16:49:50 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: <20210801234950.GB3388@mcvoy.com> On Sun, Aug 01, 2021 at 07:36:53PM -0400, John Cowan wrote: > Nowadays it's a question whether fork() makes sense any more. "A fork() > in the road" [Baumann et al. 2019] < > https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf> > is an interesting argument against fork(): > > * It doesn't compose. > * It is insecure by default. > * It is slow (there are about 25 properties a process has in addition to > its memory and hardware state, and each of these needs to be copied or not) > even using COW (which is itself a Good Thing and can and should be provided > separately) > * It is incompatible with a single-address-space design. > > In short, spawn() beats fork() like a drum, and fork() should be > deprecated. To be sure, the paper comes out of Microsoft Research, but I > find it pretty compelling anyway. When we were working on supporting BitKeeper on Windows, MacOS, all the various Unix versions, and Linux, we implemented all the needed libc stuff on Windows (so we could pretend we were not running on Windows). Everything except fork(), we made a spawnvp() interface. That's the one thing that made more sense than the Unix way. I have called fork() directly in decades. From crossd at gmail.com Mon Aug 2 09:58:50 2021 From: crossd at gmail.com (Dan Cross) Date: Sun, 1 Aug 2021 19:58:50 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On Sun, Aug 1, 2021 at 7:37 PM John Cowan wrote: > On Sun, Aug 1, 2021 at 5:55 PM Dan Cross wrote: > >> Looking at other systems that were available roughly around the time of >> Unix (TENEX, Multics), it strikes me that the Unix was a bit of an odd-duck >> with the way it handled exec in terms of destructively overlaying the >> memory of the user portion of a process with a new image; am I wrong here? >> > > See dmr's paper at for > details, but in short exec and its equivalents elsewhere have always > overlaid the running program with another program. > That's a great paper and I've really enjoyed revisiting it over the years, but while it does a great job of explaining how the Unix mechanism worked, and touches on the "why", it doesn't contrast with other schemes. I suppose my question could be rephrased as, if the early Unix implementers had had more resources to work with, would they have chosen a model more along the lines used by Multics and Twenex, or would they have elected to do basically what they did? That's probably impossible to answer, but gets at what they thought about how other systems operated. Early versions of PDP-7 Linux used the same process model as Tenex: one > process per terminal which alternated between running the shell and a user > program. So exec() loaded the user program on top of the shell. Indeed, > this wasn't even a syscall; the shell itself wrote a tiny program loader > into the top of memory that read the new program, which was open for > reading, and jumped to it. Likewise, exit() was a specialized exec() that > reloaded the shell. The Tenex and Multics shells had more memory to play > with and didn't have to use these self-overlaying tricks[*]: they loaded > your program into available memory and called it as a subroutine, which > accounts for the name "shell". > Presumably the virtual memory hardware could also be used to protect the shell from a malicious or errant program trashing the image of the shell in memory. [snip] > Nowadays it's a question whether fork() makes sense any more. "A fork() > in the road" [Baumann et al. 2019] < > https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf> > is an interesting argument against fork(): > > * It doesn't compose. > * It is insecure by default. > * It is slow (there are about 25 properties a process has in addition to > its memory and hardware state, and each of these needs to be copied or not) > even using COW (which is itself a Good Thing and can and should be provided > separately) > * It is incompatible with a single-address-space design. > > In short, spawn() beats fork() like a drum, and fork() should be > deprecated. To be sure, the paper comes out of Microsoft Research, but I > find it pretty compelling anyway. > Spawn vs fork/exec is a false dichotomy, though. We talked about the fork paper when it came out, and here's what I wrote about it at the time: https://minnie.tuhs.org/pipermail/tuhs/2019-April/017700.html [*] My very favorite self-overlaying program was the PDP-8 bootstrap for > the DF32 disk drive. You toggled in two instructions at locations 30 and > 31 meaning "load disk registers and go" and "jump to self" respectively, > hit the Clear key on the front panel, which cleared all registers, and > started up at 30. > > The first instruction told the disk to start reading sector 0 of the disk > into location 0 in memory (because all the registers were 0, including the > disk instruction register where 0 = READ) and the second instruction kept > the CPU busy waiting. As the sector loaded, the two instructions were > overwritten by "skip if disk ready" and "jump to previous address", which > would wait until the whole sector was loaded. Then the OS could be loaded > using the primitive disk driver in block 0. > Very nice; that's highly reminiscent of a Sergeant-style forth: https://pygmy.utoh.org/3ins4th.html One wonders if the PDP-8 was one of Sergeant's inspirations? - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreww591 at gmail.com Mon Aug 2 10:13:18 2021 From: andreww591 at gmail.com (Andrew Warkentin) Date: Sun, 1 Aug 2021 18:13:18 -0600 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On 8/1/21, John Cowan wrote: > > Nowadays it's a question whether fork() makes sense any more. "A fork() > in the road" [Baumann et al. 2019] < > https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf> > is an interesting argument against fork(): > > * It doesn't compose. > * It is insecure by default. > * It is slow (there are about 25 properties a process has in addition to > its memory and hardware state, and each of these needs to be copied or not) > even using COW (which is itself a Good Thing and can and should be provided > separately) > * It is incompatible with a single-address-space design. > > In short, spawn() beats fork() like a drum, and fork() should be > deprecated. To be sure, the paper comes out of Microsoft Research, but I > find it pretty compelling anyway. > There's a third kind of primitive that is superior to either spawn() or fork() IMO, specifically one that creates a completely empty child process and returns a context that lets the parent set up the child's state using normal APIs. To start the child the parent would either call exec() to start the child running a different program, or call a new function that starts the child with a parent-provided entry point and whatever memory mappings the parent set up. Both fork() and spawn() could be implemented on top of this easily enough with basically no additional overhead compared to implementing both as primitives. This is what I plan to do on the OS I'm writing (manipulating the child's state won't require any additional primitives beyond regular file I/O since literally all process state will have a file-based interface). From cowan at ccil.org Mon Aug 2 10:18:22 2021 From: cowan at ccil.org (John Cowan) Date: Sun, 1 Aug 2021 20:18:22 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On Sun, Aug 1, 2021 at 8:13 PM Andrew Warkentin wrote To start the child the parent would either > call exec() to start the child running a different program, or call a > new function that starts the child with a parent-provided entry point > and whatever memory mappings the parent set up. > This is what I plan to do on the OS I'm writing > (manipulating the child's state won't require any additional > primitives beyond regular file I/O since literally all process state > will have a file-based interface). > In that case you don't need *any* primitive except create_empty_process(): you can do exec() by opening the file, writing to /proc//mem and then to /pc-and-go. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lm at mcvoy.com Mon Aug 2 10:28:45 2021 From: lm at mcvoy.com (Larry McVoy) Date: Sun, 1 Aug 2021 17:28:45 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <20210801234950.GB3388@mcvoy.com> References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> <20210801234950.GB3388@mcvoy.com> Message-ID: <20210802002845.GC3388@mcvoy.com> On Sun, Aug 01, 2021 at 04:49:50PM -0700, Larry McVoy wrote: > On Sun, Aug 01, 2021 at 07:36:53PM -0400, John Cowan wrote: > > Nowadays it's a question whether fork() makes sense any more. "A fork() > > in the road" [Baumann et al. 2019] < > > https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf> > > is an interesting argument against fork(): > > > > * It doesn't compose. > > * It is insecure by default. > > * It is slow (there are about 25 properties a process has in addition to > > its memory and hardware state, and each of these needs to be copied or not) > > even using COW (which is itself a Good Thing and can and should be provided > > separately) > > * It is incompatible with a single-address-space design. > > > > In short, spawn() beats fork() like a drum, and fork() should be > > deprecated. To be sure, the paper comes out of Microsoft Research, but I > > find it pretty compelling anyway. > > When we were working on supporting BitKeeper on Windows, MacOS, all the > various Unix versions, and Linux, we implemented all the needed libc > stuff on Windows (so we could pretend we were not running on Windows). > Everything except fork(), we made a spawnvp() interface. That's the > one thing that made more sense than the Unix way. I have called fork() > directly in decades. s/have/have not/ called fork().... Sigh. -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From usotsuki at buric.co Mon Aug 2 10:29:54 2021 From: usotsuki at buric.co (Steve Nickolas) Date: Sun, 1 Aug 2021 20:29:54 -0400 (EDT) Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On Sun, 1 Aug 2021, Dan Cross wrote: > Spawn vs fork/exec is a false dichotomy, though. We talked about the fork > paper when it came out, and here's what I wrote about it at the time: > https://minnie.tuhs.org/pipermail/tuhs/2019-April/017700.html I've often wished I could run a free/open Bourne-type shell on 16-bit MS-DOS and OS/2. Porting to the former is next to impossible because of the lack of *any* concept of multitasking. Porting to the latter is difficult because multitasking isn't done anything like the Unix way. I actually like the spawn* functions better, though I think on Unix fork/exec is the most natural way to implement them. -uso. From andreww591 at gmail.com Mon Aug 2 10:54:17 2021 From: andreww591 at gmail.com (Andrew Warkentin) Date: Sun, 1 Aug 2021 18:54:17 -0600 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On 8/1/21, John Cowan wrote: > > In that case you don't need *any* primitive except create_empty_process(): > you can do exec() by opening the file, writing to /proc//mem and > then to /pc-and-go. > Yes, although that would break if the permissions for the program are execute-only (which admittedly is of limited security value in most cases). From tytso at mit.edu Mon Aug 2 11:05:03 2021 From: tytso at mit.edu (Theodore Ts'o) Date: Sun, 1 Aug 2021 21:05:03 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On Sun, Aug 01, 2021 at 06:13:18PM -0600, Andrew Warkentin wrote: > There's a third kind of primitive that is superior to either spawn() > or fork() IMO, specifically one that creates a completely empty child > process and returns a context that lets the parent set up the child's > state using normal APIs. I've seen this argument a number of times, but what's never been clear to me is what *would* the "normal APIs" be which would allow a parent to set up the child's state? How would that be accomplished? Lots of new system calls? Magic files in /proc//XXX which get manipulated somehow? (How, exactly, does one affect the child's memory map via magic read/write calls to /proc//XXX.... How about environment variables, etc.) And what are the access rights by which a process gets to reach out and touch another process's environment? Is it only allowed only for child processes? And is it only allowed before the child starts running? What if the child process is going to be running a setuid or setgid executable? The phrase "all process state will have a file-based interface" sounds good on paper, but I think it remains to be seen how well a "echo XXX > /proc//magic-file" API would actually work. The devil is really in the details.... - Ted From crossd at gmail.com Mon Aug 2 11:04:46 2021 From: crossd at gmail.com (Dan Cross) Date: Sun, 1 Aug 2021 21:04:46 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On Sun, Aug 1, 2021 at 8:19 PM John Cowan wrote: > On Sun, Aug 1, 2021 at 8:13 PM Andrew Warkentin > wrote > >> To start the child the parent would either >> call exec() to start the child running a different program, or call a >> new function that starts the child with a parent-provided entry point >> and whatever memory mappings the parent set up. > > > >> This is what I plan to do on the OS I'm writing >> (manipulating the child's state won't require any additional >> primitives beyond regular file I/O since literally all process state >> will have a file-based interface). >> > > In that case you don't need *any* primitive except create_empty_process(): > you can do exec() by opening the file, writing to /proc//mem and > then to /pc-and-go. > Sadly, that's not _quite_ true. You still need some primordial way to get the system going. Once you have a proc_create and a make_proc_runnable system call, it seems like it opens the door to doing all kinds of cool things like moving binary parsers out of the kernel and into user space, but consider how `init` gets bootstrapped: often, there's a handful of instructions that basically invoke `execl("/bin/init", "init", 0);` that you compile into the kernel; on creation of process 1, the kernel copies those instructions into a page somewhere in user portion of the address space and "returns" to it; the process then invokes /bin/init which carries on with bringing up the rest of the system. Now you're confronted with two choices: you either put a much more elaborate bootstrap into the kernel (in this day and age, probably not that hard), or you have a minimal bootstrap that's smart enough to load a smarter bootstrap that in turn can load something like init. I suppose a third option is to compile `init` in some dead simple way that you can load in the kernel as a special case, and invoke that. This problem isn't insurmountable, but it's a wide design space, and it's not quite as straight-forward as it first appears. As I mentioned in the email I linked to earlier, Akaros implemented the proc_create/proc_run model. It really was superior to fork()/exec() and I would argue superior to spawn() as well. - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreww591 at gmail.com Mon Aug 2 12:10:46 2021 From: andreww591 at gmail.com (Andrew Warkentin) Date: Sun, 1 Aug 2021 20:10:46 -0600 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On 8/1/21, Theodore Ts'o wrote: > > I've seen this argument a number of times, but what's never been clear > to me is what *would* the "normal APIs" be which would allow a parent > to set up the child's state? How would that be accomplished? Lots of > new system calls? Magic files in /proc//XXX which get > manipulated somehow? (How, exactly, does one affect the child's > memory map via magic read/write calls to /proc//XXX.... How > about environment variables, etc.) > My OS will be microkernel-based and even the RPC channel to the VFS itself will be a file (with some special semantics). read(), write() and seek() will bypass the VFS entirely and call the kernel to directly communicate with the destination process. The call to create an empty process will return a new RPC channel and there will be an API to temporarily switch to an alternate channel so that VFS calls occur in the child context instead of the parent. All process memory, even the heap and stack, will be implemented as memory-mapped files in a per-process filesystem under /proc/. This will be a special "shadowfs" that allows creating files that shadow ranges of other files (either on disk or in memory). Environment variables will also be exposed in /proc of course. > > And what are the access rights by which a process gets to reach out > and touch another process's environment? Is it only allowed only for > child processes? And is it only allowed before the child starts > running? What if the child process is going to be running a setuid or > setgid executable? > Any process that has permissions to access the RPC channel file and memory mapping shadow files in /proc/ will be able to manipulate the state. The RPC channel will cease to function after the child has been started. setuid and setgid executables will not be supported at all (there will instead be a role-based access control system layered on top of a per-process file permission list, which will allow privilege escalation on exec in certain situations defined by configuration). > > The phrase "all process state will have a file-based interface" sounds > good on paper, but I think it remains to be seen how well a "echo XXX >> /proc//magic-file" API would actually work. The devil is > really in the details.... > Even though everything will use a file-based implementation underneath, there will be a utility library layered on top of it so that user code doesn't have to contain lots of open()-read()/write()-close() boilerplate. From bakul at iitbombay.org Mon Aug 2 12:32:54 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Sun, 1 Aug 2021 19:32:54 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: On Aug 1, 2021, at 6:05 PM, Theodore Ts'o wrote: > > On Sun, Aug 01, 2021 at 06:13:18PM -0600, Andrew Warkentin wrote: >> There's a third kind of primitive that is superior to either spawn() >> or fork() IMO, specifically one that creates a completely empty child >> process and returns a context that lets the parent set up the child's >> state using normal APIs. > > I've seen this argument a number of times, but what's never been clear > to me is what *would* the "normal APIs" be which would allow a parent > to set up the child's state? How would that be accomplished? Lots of > new system calls? Magic files in /proc//XXX which get > manipulated somehow? (How, exactly, does one affect the child's > memory map via magic read/write calls to /proc//XXX.... How > about environment variables, etc.) > > And what are the access rights by which a process gets to reach out > and touch another process's environment? Is it only allowed only for > child processes? And is it only allowed before the child starts > running? What if the child process is going to be running a setuid or > setgid executable? From the "KeyKOS Nanokernel Architecture" (1992) paper: ---- KeyKOS processes are created by building a segment that will become the program address space, obtaining a fresh domain, and inserting the segment key in the domain's address slot. The domain is created in the waiting state, which means that it is waiting for a message. A threads paradigm can be supported by having two or more domains share a common address space segment. Because domain initialization is such a common operation, KeyKOS provides a mechanism to generate "prepackaged" domains. A factory is an entity that constructs other domains. Every factory creates a particular type of domain. For example, the queue factory creates domains that provide queuing services. An important aspect of factories is the ability of the client to determine their trustworthiness. It is possible for a client to determine whether an object created by a factory is secure. ---- This paper also talks about their attempt to emulate Unix on top. http://css.csail.mit.edu/6.858/2009/readings/keykos.pdf From jim at deitygraveyard.com Mon Aug 2 12:34:39 2021 From: jim at deitygraveyard.com (Jim Carpenter) Date: Sun, 1 Aug 2021 22:34:39 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: On Sat, Jul 31, 2021 at 1:08 PM Ron Young wrote: > FYI, There is also a unix version of the COMND JSYS capability. It was > developed at Columbia University as part of their "mm" mail manager. It > is located in to the ccmd subdirectory in the mm.tar.gz file. > > url: https://www.kermitproject.org/mm/ > > ftp://ftp.kermitproject.org/kermit/mm/mm.tar.gz > I hope there are copies of these files. Looks like that FTP server has some major corruption going on. The MM files are inaccessible, as are others. Jim From rly1 at embarqmail.com Mon Aug 2 12:38:24 2021 From: rly1 at embarqmail.com (Ron Young) Date: Sun, 01 Aug 2021 19:38:24 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: Message-ID: <5630b4bc-be3f-4a7b-ba6f-9619e0cc4d4d@email.android.com> An HTML attachment was scrubbed... URL: From douglas.mcilroy at dartmouth.edu Mon Aug 2 12:42:53 2021 From: douglas.mcilroy at dartmouth.edu (Douglas McIlroy) Date: Sun, 1 Aug 2021 22:42:53 -0400 Subject: [TUHS] Systematic approach to command-line interfaces Message-ID: > spawn() beats fork()[;] fork() should be deprecated Spawn is a further complication of exec, which tells what signals and file descriptors to inherit in addition to what arguments and environment variables to pass. Fork has a place. For example, Program 1 in www.cs.dartmouth.edu/~doug/sieve/sieve.pdf forks like crazy and never execs. To use spawn, the program would have to be split in three (or be passed a switch setting). While you may dismiss Program 1 as merely a neat demo, the same idea applies in parallelizing code for use in a multiprocessor world. Doug From steffen at sdaoden.eu Mon Aug 2 22:11:46 2021 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Mon, 02 Aug 2021 14:11:46 +0200 Subject: [TUHS] Systematic approach to command-line interfaces [ meta issues ] In-Reply-To: <202108011948.171JmAcK001895@freefriends.org> References: <202108011948.171JmAcK001895@freefriends.org> Message-ID: <20210802121146.mlcPX%steffen@sdaoden.eu> arnold at skeeve.com wrote in <202108011948.171JmAcK001895 at freefriends.org>: |Douglas McIlroy wrote: | |> In realizing the white paper's desire to "have the parser |> provide the results to the program", it's likely that the mechanism |> will, like Yacc, go beyond parsing and invoke semantic actions |> as it identifies tree nodes. | |I have to admit that all this feels like overkill. Parsing options |is only a very small part of the real work that a program does. | |Speaking for myself, I want something simple and regular that will |get the job done and let me get on with the actual business of |my software. A grammar just for command-line argument parsing feels |like the tail wagging the dog: not nearly enough ROI, at least |for me. | |I happen to like the getopt_long interface designed by the GNU |project. It's easy to learn, setup and use. Once it's in place |it's set and forget. By coincidence just last week i stumbled over (actually searched and fixed) an issue where that terrible command line resorting hit me where i did not expect it. Ie after changing aspects of a scripts that affect variable content, where that string then appended to a string constant and then evaluated-passed to a program, where the variable content did never contain a hyphen-minus initially, but after the rewrite. So they saw a leading hyphen-minus somewhere on the line and turned it into an option. (The fix was easy, just turn 'X'$Y into 'X'"$Y", it maybe should have always been written like that, but it seemed unnecessary at first.) |My two cents, For C++/C i have always had my own one which can long options, optionally relates long to short options, where the long ones also can include a help string (all in one string, as in "debug;d;" N_("identical to -Sdebug") and N_() expands to literal). I agree with the other post that turning command lines into a tree of nodes is great, but of course this is hard to define. For first level only yet (without support for multiplexer commands, ie, commands where the first command chooses an actual command) i have this for the mailer i maintain, for some commands already. It is a pain to write things like the following by hand mx_CMD_ARG_DESC_SUBCLASS_DEF(call, 2, a_cmd_cad_call){ {mx_CMD_ARG_DESC_SHEXP | mx_CMD_ARG_DESC_HONOUR_STOP, n_SHEXP_PARSE_TRIM_IFSSPACE}, /* macro name */ {mx_CMD_ARG_DESC_SHEXP | mx_CMD_ARG_DESC_OPTION | mx_CMD_ARG_DESC_GREEDY | mx_CMD_ARG_DESC_HONOUR_STOP, n_SHEXP_PARSE_IFS_VAR | n_SHEXP_PARSE_TRIM_IFSSPACE} /* args */ }mx_CMD_ARG_DESC_SUBCLASS_DEF_END; --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From tytso at mit.edu Tue Aug 3 00:58:50 2021 From: tytso at mit.edu (Theodore Ts'o) Date: Mon, 2 Aug 2021 10:58:50 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: On Sun, Aug 01, 2021 at 10:42:53PM -0400, Douglas McIlroy wrote: > > spawn() beats fork()[;] fork() should be deprecated > > Spawn is a further complication of exec, which tells what signals and > file descriptors to inherit in addition to what arguments and > environment variables to pass. > > Fork has a place. For example, Program 1 in > www.cs.dartmouth.edu/~doug/sieve/sieve.pdf forks like crazy and never > execs. To use spawn, the program would have to be split in three (or > be passed a switch setting). > > While you may dismiss Program 1 as merely a neat demo, the same idea > applies in parallelizing code for use in a multiprocessor world. It's certainly clear that some kind of primitive is needed to create new threads. An open question is whether if there exists some kind of "new thread" primitve plus either spawn(2) or some kind of "create a child process and then then frob like crazy using 'echo XXX > /proc//'" whether there still is a need for a fork(2) system call. Obviously, as soon as we start going down this path, we're deviated quite strongly from the "radical simplicity" of Unix Version 7 that people have accused modern systems (whether they be Linux or FreeBSD) of lacking. It's rather interesting that we haven't heard complaints about how people who dare to try come up with new API's are somehow traitors to "The Unix Philosphy" that we've seen on other threads. :-) - Ted From lars at nocrew.org Tue Aug 3 03:33:29 2021 From: lars at nocrew.org (Lars Brinkhoff) Date: Mon, 02 Aug 2021 17:33:29 +0000 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: (Andrew Warkentin's message of "Sun, 1 Aug 2021 18:13:18 -0600") References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: <7wh7g73g5y.fsf@junk.nocrew.org> John Cowan wrote: > Andrew Warkentin wrote: > > There's a third kind of primitive that is superior to either spawn() > > or fork() IMO, specifically one that creates a completely empty > > child process and returns a context that lets the parent set up the > > child's state using normal APIs. > In that case you don't need *any* primitive except create_empty_process(): > you can do exec() by opening the file, writing to /proc//mem That's almost exactly what what ITS does. You open the USR: device and get a file descriptor (not really, but close enough) into the child process (inferior job). From lars at nocrew.org Tue Aug 3 03:37:20 2021 From: lars at nocrew.org (Lars Brinkhoff) Date: Mon, 02 Aug 2021 17:37:20 +0000 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: (John Cowan's message of "Sun, 1 Aug 2021 19:36:53 -0400") References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> Message-ID: <7wczqv3fzj.fsf@junk.nocrew.org> John Cowan wrote: > Early versions of PDP-7 [Unix] used the same process model as Tenex I understand both Tenex and Unix got the concept of "fork" from Project Genie. From athornton at gmail.com Tue Aug 3 04:15:34 2021 From: athornton at gmail.com (Adam Thornton) Date: Mon, 2 Aug 2021 11:15:34 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: It's a measure of Unix having been wounded by its own success. fork() is a great model for a single-threaded text processing pipeline to do automated typesetting. (More generally, anything that is a straightforward composition of filter/transform stages.) Which is, y'know, what Unix is *for*. It's not so great for a responsive GUI in front of a multi-function interactive program. These days, the vast majority of Unix applications are "stuff people play with on their phones." Adam On Mon, Aug 2, 2021 at 7:59 AM Theodore Ts'o wrote: > On Sun, Aug 01, 2021 at 10:42:53PM -0400, Douglas McIlroy wrote: > > > spawn() beats fork()[;] fork() should be deprecated > > > > Spawn is a further complication of exec, which tells what signals and > > file descriptors to inherit in addition to what arguments and > > environment variables to pass. > > > > Fork has a place. For example, Program 1 in > > www.cs.dartmouth.edu/~doug/sieve/sieve.pdf forks like crazy and never > > execs. To use spawn, the program would have to be split in three (or > > be passed a switch setting). > > > > While you may dismiss Program 1 as merely a neat demo, the same idea > > applies in parallelizing code for use in a multiprocessor world. > > It's certainly clear that some kind of primitive is needed to create > new threads. An open question is whether if there exists some kind of > "new thread" primitve plus either spawn(2) or some kind of "create a > child process and then then frob like crazy using 'echo XXX > > /proc//'" whether there still is a need for a > fork(2) system call. > > Obviously, as soon as we start going down this path, we're deviated > quite strongly from the "radical simplicity" of Unix Version 7 that > people have accused modern systems (whether they be Linux or FreeBSD) > of lacking. It's rather interesting that we haven't heard complaints > about how people who dare to try come up with new API's are somehow > traitors to "The Unix Philosphy" that we've seen on other threads. :-) > > - Ted > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imp at bsdimp.com Tue Aug 3 04:24:05 2021 From: imp at bsdimp.com (Warner Losh) Date: Mon, 2 Aug 2021 12:24:05 -0600 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: On Mon, Aug 2, 2021 at 12:16 PM Adam Thornton wrote: > It's a measure of Unix having been wounded by its own success. > > fork() is a great model for a single-threaded text processing pipeline to > do automated typesetting. (More generally, anything that is a > straightforward composition of filter/transform stages.) Which is, y'know, > what Unix is *for*. > fork() dates from a time that demand paging wasn't a thing. Processes were as cheap as it got. There were no threads. All the different variations on a theme on fork() since then have been to either make threads super cheap to create, to optimize the exec case (which already has been discussed a bit:), and/or to control what the new process inherits. It's not so great for a responsive GUI in front of a multi-function > interactive program. > > These days, the vast majority of Unix applications are "stuff people play > with on their phones." > Ah, a thread-heavy environment that's not all that exec intensive (but that's complicated enough you can no longer safely do a naive fork/exec when you need to)... But mostly, it's threads. Warner > Adam > > On Mon, Aug 2, 2021 at 7:59 AM Theodore Ts'o wrote: > >> On Sun, Aug 01, 2021 at 10:42:53PM -0400, Douglas McIlroy wrote: >> > > spawn() beats fork()[;] fork() should be deprecated >> > >> > Spawn is a further complication of exec, which tells what signals and >> > file descriptors to inherit in addition to what arguments and >> > environment variables to pass. >> > >> > Fork has a place. For example, Program 1 in >> > www.cs.dartmouth.edu/~doug/sieve/sieve.pdf forks like crazy and never >> > execs. To use spawn, the program would have to be split in three (or >> > be passed a switch setting). >> > >> > While you may dismiss Program 1 as merely a neat demo, the same idea >> > applies in parallelizing code for use in a multiprocessor world. >> >> It's certainly clear that some kind of primitive is needed to create >> new threads. An open question is whether if there exists some kind of >> "new thread" primitve plus either spawn(2) or some kind of "create a >> child process and then then frob like crazy using 'echo XXX > >> /proc//'" whether there still is a need for a >> fork(2) system call. >> >> Obviously, as soon as we start going down this path, we're deviated >> quite strongly from the "radical simplicity" of Unix Version 7 that >> people have accused modern systems (whether they be Linux or FreeBSD) >> of lacking. It's rather interesting that we haven't heard complaints >> about how people who dare to try come up with new API's are somehow >> traitors to "The Unix Philosphy" that we've seen on other threads. :-) >> >> - Ted >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Tue Aug 3 04:52:55 2021 From: clemc at ccc.com (Clem Cole) Date: Mon, 2 Aug 2021 14:52:55 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <7wczqv3fzj.fsf@junk.nocrew.org> References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> <7wczqv3fzj.fsf@junk.nocrew.org> Message-ID: On Mon, Aug 2, 2021 at 1:37 PM Lars Brinkhoff wrote: > John Cowan wrote: > > Early versions of PDP-7 [Unix] used the same process model as Tenex > > I understand both Tenex and Unix got the concept of "fork" from Project > Genie. Should be required reading of all intro to OS students: Programming Semantics for Multiprogrammed Computations Jack B. Dennis and Earl C. Van Horn Massachusetts Institute of Technology, Cambridge, Massachusetts Volume 9 / Number 3 / March, 1966 [If your Internet search fails you and/or you find it behind an ACM paywall or the like, drop me a line, I'll forward a PDF of a scan]. ᐧ ᐧ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cowan at ccil.org Tue Aug 3 06:55:32 2021 From: cowan at ccil.org (John Cowan) Date: Mon, 2 Aug 2021 16:55:32 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: On Mon, Aug 2, 2021 at 2:16 PM Adam Thornton wrote: > fork() is a great model for a single-threaded text processing pipeline to > do automated typesetting. (More generally, anything that is a > straightforward composition of filter/transform stages.) Which is, y'know, > what Unix is *for*. > Indeed. But it's also a very good model for "baking" web pages in the background so that you can serve them up with a plain dumb web server, maybe with a bit of JS to provide some auto-updating, especially if the source data is stored not in a database but in the file system. The result is a page that displays (modulo network latency) as fast as you can hit the Enter key in the address bar. (The weak point is the lack of dependency management when the system is too big to rebake all the pages each time. Perhaps make(1), which Alex Shinn described as "a beautiful little Prolog for the file system", is the Right Thing.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From cowan at ccil.org Tue Aug 3 06:59:39 2021 From: cowan at ccil.org (John Cowan) Date: Mon, 2 Aug 2021 16:59:39 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> <7wczqv3fzj.fsf@junk.nocrew.org> Message-ID: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.124.6859&rep=rep1&type=pdf works fine, no paywall. On Mon, Aug 2, 2021 at 2:53 PM Clem Cole wrote: > > On Mon, Aug 2, 2021 at 1:37 PM Lars Brinkhoff wrote: > >> John Cowan wrote: >> > Early versions of PDP-7 [Unix] used the same process model as Tenex >> >> I understand both Tenex and Unix got the concept of "fork" from Project >> Genie. > > > Should be required reading of all intro to OS students: > > Programming Semantics for Multiprogrammed Computations > > Jack B. Dennis and Earl C. Van Horn > > Massachusetts Institute of Technology, Cambridge, Massachusetts > > Volume 9 / Number 3 / March, 1966 > > [If your Internet search fails you and/or you find it behind an ACM > paywall or the like, drop me a line, I'll forward a PDF of a scan]. > > > ᐧ > ᐧ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aek at bitsavers.org Tue Aug 3 07:06:52 2021 From: aek at bitsavers.org (Al Kossow) Date: Mon, 2 Aug 2021 14:06:52 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> <7wczqv3fzj.fsf@junk.nocrew.org> Message-ID: <64befcee-d10f-7880-595e-6a53558cc39d@bitsavers.org> On 8/2/21 1:59 PM, John Cowan wrote: > I understand both Tenex and Unix got the concept of "fork" from Project > Genie. Original UCB source material on Genie can be found at bitsavers. From clemc at ccc.com Tue Aug 3 07:13:54 2021 From: clemc at ccc.com (Clem Cole) Date: Mon, 2 Aug 2021 17:13:54 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> <7wczqv3fzj.fsf@junk.nocrew.org> Message-ID: Sorry bad cut/paste -- that was from CACM if its was not obvious ᐧ On Mon, Aug 2, 2021 at 2:52 PM Clem Cole wrote: > > On Mon, Aug 2, 2021 at 1:37 PM Lars Brinkhoff wrote: > >> John Cowan wrote: >> > Early versions of PDP-7 [Unix] used the same process model as Tenex >> >> I understand both Tenex and Unix got the concept of "fork" from Project >> Genie. > > > Should be required reading of all intro to OS students: > > Programming Semantics for Multiprogrammed Computations > > Jack B. Dennis and Earl C. Van Horn > > Massachusetts Institute of Technology, Cambridge, Massachusetts > > Volume 9 / Number 3 / March, 1966 > > [If your Internet search fails you and/or you find it behind an ACM > paywall or the like, drop me a line, I'll forward a PDF of a scan]. > > > ᐧ > ᐧ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clemc at ccc.com Tue Aug 3 07:14:40 2021 From: clemc at ccc.com (Clem Cole) Date: Mon, 2 Aug 2021 17:14:40 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <20210731142533.69caf929@moon> <40763c2d-52ad-eb01-8bf8-85acf6fee700@case.edu> <7wczqv3fzj.fsf@junk.nocrew.org> Message-ID: excellent ᐧ On Mon, Aug 2, 2021 at 4:59 PM John Cowan wrote: > > https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.124.6859&rep=rep1&type=pdf > works fine, no paywall. > > On Mon, Aug 2, 2021 at 2:53 PM Clem Cole wrote: > >> >> On Mon, Aug 2, 2021 at 1:37 PM Lars Brinkhoff wrote: >> >>> John Cowan wrote: >>> > Early versions of PDP-7 [Unix] used the same process model as Tenex >>> >>> I understand both Tenex and Unix got the concept of "fork" from Project >>> Genie. >> >> >> Should be required reading of all intro to OS students: >> >> Programming Semantics for Multiprogrammed Computations >> >> Jack B. Dennis and Earl C. Van Horn >> >> Massachusetts Institute of Technology, Cambridge, Massachusetts >> >> Volume 9 / Number 3 / March, 1966 >> >> [If your Internet search fails you and/or you find it behind an ACM >> paywall or the like, drop me a line, I'll forward a PDF of a scan]. >> >> >> ᐧ >> ᐧ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon at fourwinds.com Tue Aug 3 07:06:18 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Mon, 02 Aug 2021 14:06:18 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: <202108022106.172L6I1i135429@darkstar.fourwinds.com> John Cowan writes: > > > fork() is a great model for a single-threaded text processing pipeline to > > do automated typesetting. (More generally, anything that is a > > straightforward composition of filter/transform stages.) Which is, y'know, > > what Unix is *for*. > > > > Indeed. But it's also a very good model for "baking" web pages in the > background so that you can serve them up with a plain dumb web server, > maybe with a bit of JS to provide some auto-updating, especially if the > source data is stored not in a database but in the file system. The result > is a page that displays (modulo network latency) as fast as you can hit the > Enter key in the address bar. > > (The weak point is the lack of dependency management when the system is too > big to rebake all the pages each time. Perhaps make(1), which Alex Shinn > described as "a beautiful little Prolog for the file system", is the Right > Thing.) We have, of course, had similar discussions many times on this list. I think that the root issue is the false equivalence of "I don't understand this well enough to be able to use it effectively to solve my problem" with "it's broken/obsolete/dated". From crossd at gmail.com Tue Aug 3 07:25:47 2021 From: crossd at gmail.com (Dan Cross) Date: Mon, 2 Aug 2021 17:25:47 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <202108022106.172L6I1i135429@darkstar.fourwinds.com> References: <202108022106.172L6I1i135429@darkstar.fourwinds.com> Message-ID: On Mon, Aug 2, 2021 at 5:19 PM Jon Steinhart wrote: > John Cowan writes: > > > fork() is a great model for a single-threaded text processing pipeline > to > > > do automated typesetting. (More generally, anything that is a > > > straightforward composition of filter/transform stages.) Which is, > y'know, > > > what Unix is *for*. > > > > > > > Indeed. But it's also a very good model for "baking" web pages in the > > background so that you can serve them up with a plain dumb web server, > > maybe with a bit of JS to provide some auto-updating, especially if the > > source data is stored not in a database but in the file system. The > result > > is a page that displays (modulo network latency) as fast as you can hit > the > > Enter key in the address bar. > > > > (The weak point is the lack of dependency management when the system is > too > > big to rebake all the pages each time. Perhaps make(1), which Alex Shinn > > described as "a beautiful little Prolog for the file system", is the > Right > > Thing.) > > We have, of course, had similar discussions many times on this list. > I think that the root issue is the false equivalence of "I don't > understand this well enough to be able to use it effectively to solve > my problem" with "it's broken/obsolete/dated". > That's a bit unfair. One can understand something and see value in it and still appreciate its limitations. Fork has served us well for more than five decades; I've got no argument with that. However, should we never question whether it continues to be the right, or best, abstraction as the environment around it continues to evolve? - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon at fourwinds.com Tue Aug 3 07:59:38 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Mon, 02 Aug 2021 14:59:38 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <202108022106.172L6I1i135429@darkstar.fourwinds.com> Message-ID: <202108022159.172LxcDR185552@darkstar.fourwinds.com> Dan Cross writes: > That's a bit unfair. One can understand something and see value in it and > still appreciate its limitations. > > Fork has served us well for more than five decades; I've got no argument > with that. However, should we never question whether it continues to be the > right, or best, abstraction as the environment around it continues to > evolve? Oh, sorry, wasn't meaning to be categorical there. Main reason that it came to mind was John's web example; many have said in the past that the UNIX model couldn't do that until they figured out that it actually could. From cowan at ccil.org Tue Aug 3 08:33:30 2021 From: cowan at ccil.org (John Cowan) Date: Mon, 2 Aug 2021 18:33:30 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <202108022159.172LxcDR185552@darkstar.fourwinds.com> References: <202108022106.172L6I1i135429@darkstar.fourwinds.com> <202108022159.172LxcDR185552@darkstar.fourwinds.com> Message-ID: On Mon, Aug 2, 2021 at 6:00 PM Jon Steinhart wrote:> > Oh, sorry, wasn't meaning to be categorical there. Main reason that it > came > to mind was John's web example; many have said in the past that the UNIX > model couldn't do that until they figured out that it actually could. > I sorta lost track of what I was saying there: spawn*() would work fine in pipelines, since they involve fork-quickly-followed-by-exec. Doug's nifty sieve example, on the other hand, would not: the Right Thing there is Go, or else goroutines in C (either libmill or its successor libdill as you prefer) since the sieve doesn't actually involve any sort of global state for which processes are relevant. Granted, the C libraries have a little bit of x86_64 asm in them (but you are not expected to understand this). -------------- next part -------------- An HTML attachment was scrubbed... URL: From douglas.mcilroy at dartmouth.edu Tue Aug 3 09:38:42 2021 From: douglas.mcilroy at dartmouth.edu (Douglas McIlroy) Date: Mon, 2 Aug 2021 19:38:42 -0400 Subject: [TUHS] Why does refer(1) have no database field for "edition"?. Message-ID: Another issue I have run into is recursion, when a reference includes another reference. This comes up in various forms: Also published as ... Errata available at ... Summarized [or reviewed] in ... Preprint available at ... Often such a reference takes the form of a URL or a page in a journal or in a proceedings. This can be most succinctly placed in situ -- formatted consistently with other references. If the reference identifies an alternate mode of publication, it may lack %A or %T fields. Partial proposal. a %O field may contain a reference, with no further recursion, The contained reference will be formatted in line in the %O text unless references are accumulated and the contained reference is not unique. Doug From bakul at iitbombay.org Tue Aug 3 10:20:41 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Mon, 2 Aug 2021 17:20:41 -0700 Subject: [TUHS] fork (Re: Systematic approach to command-line interfaces In-Reply-To: References: <202108022106.172L6I1i135429@darkstar.fourwinds.com> Message-ID: On Aug 2, 2021, at 2:25 PM, Dan Cross wrote: > > Fork has served us well for more than five decades; I've got no argument with that. However, should we never question whether it continues to be the right, or best, abstraction as the environment around it continues to evolve? An os with no fork() can be used to reimplement fork() for backward compatibility. You will anyway need all the things fork() used to do in this brave new world. For example, you will need to specify which signals and file descriptors to inherit. Something like: jmp_buf state; pid = 0; if (!setjmp(state) && !pid) { fd = openAt(hostfd, "/n/address-space/new", ...); <> pid = breathe_life(fd, state, signals, fds); longjmp(state, 1); return pid; } The /n/address-space filesystem yields a new address space on a given host (as represented by the hostfd). You can then "copy" to it to fill it up from the current proc's memory. Copy should be done using COW if the new proc is on the same host (or even remote but that opens up another area of discussion...). breathe_life is given where everything is set up and it just has to create a new thread, associated with the address, wire up signals as file descriptors that are to be inherited and arrange things so that on return from syscall in the child, process state will be as per "state". If signals and fds can be wired up to a remote host, this can be used to "migrate" a process. There is likely much more process state in the kernel mode which would have to be packaged up somehow. There may be other breakage if the child & parent are on different hosts. From bakul at iitbombay.org Tue Aug 3 10:21:09 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Mon, 2 Aug 2021 17:21:09 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: Or perhaps the issue is not having graphics/GUI designers with the creativity and sensibilities of the early Bell Labs crowd of researchers? I keep thinking there ought to be something simpler/more elegant than the current graphics subsystems.... > On Aug 2, 2021, at 11:16 AM, Adam Thornton wrote: > >  > It's a measure of Unix having been wounded by its own success. > > fork() is a great model for a single-threaded text processing pipeline to do automated typesetting. (More generally, anything that is a straightforward composition of filter/transform stages.) Which is, y'know, what Unix is *for*. > > It's not so great for a responsive GUI in front of a multi-function interactive program. > > These days, the vast majority of Unix applications are "stuff people play with on their phones." > > Adam > > On Mon, Aug 2, 2021 at 7:59 AM Theodore Ts'o wrote: > On Sun, Aug 01, 2021 at 10:42:53PM -0400, Douglas McIlroy wrote: > > > spawn() beats fork()[;] fork() should be deprecated > > > > Spawn is a further complication of exec, which tells what signals and > > file descriptors to inherit in addition to what arguments and > > environment variables to pass. > > > > Fork has a place. For example, Program 1 in > > www.cs.dartmouth.edu/~doug/sieve/sieve.pdf forks like crazy and never > > execs. To use spawn, the program would have to be split in three (or > > be passed a switch setting). > > > > While you may dismiss Program 1 as merely a neat demo, the same idea > > applies in parallelizing code for use in a multiprocessor world. > > It's certainly clear that some kind of primitive is needed to create > new threads. An open question is whether if there exists some kind of > "new thread" primitve plus either spawn(2) or some kind of "create a > child process and then then frob like crazy using 'echo XXX > > /proc//'" whether there still is a need for a > fork(2) system call. > > Obviously, as soon as we start going down this path, we're deviated > quite strongly from the "radical simplicity" of Unix Version 7 that > people have accused modern systems (whether they be Linux or FreeBSD) > of lacking. It's rather interesting that we haven't heard complaints > about how people who dare to try come up with new API's are somehow > traitors to "The Unix Philosphy" that we've seen on other threads. :-) > > - Ted From athornton at gmail.com Tue Aug 3 10:59:12 2021 From: athornton at gmail.com (Adam Thornton) Date: Mon, 2 Aug 2021 17:59:12 -0700 Subject: [TUHS] fork (Re: Systematic approach to command-line interfaces In-Reply-To: References: <202108022106.172L6I1i135429@darkstar.fourwinds.com> Message-ID: <44B352BC-D2A3-4AE0-B01E-2A13DB1BE816@gmail.com> > On Aug 2, 2021, at 5:20 PM, Bakul Shah wrote: > > On Aug 2, 2021, at 2:25 PM, Dan Cross wrote: >> >> Fork has served us well for more than five decades; I've got no argument with that. However, should we never question whether it continues to be the right, or best, abstraction as the environment around it continues to evolve? > > An os with no fork() can be used to reimplement fork() for backward > compatibility. You will anyway need all the things fork() used to do > in this brave new world. Oh man, now I’m having flashbacks to early Cygwin and what was that, DJGPP for OS/2? Well, I picked a lousy week to give up sniffing glue anyway. Adam From usotsuki at buric.co Tue Aug 3 11:20:15 2021 From: usotsuki at buric.co (Steve Nickolas) Date: Mon, 2 Aug 2021 21:20:15 -0400 (EDT) Subject: [TUHS] fork (Re: Systematic approach to command-line interfaces In-Reply-To: <44B352BC-D2A3-4AE0-B01E-2A13DB1BE816@gmail.com> References: <202108022106.172L6I1i135429@darkstar.fourwinds.com> <44B352BC-D2A3-4AE0-B01E-2A13DB1BE816@gmail.com> Message-ID: On Mon, 2 Aug 2021, Adam Thornton wrote: > > >> On Aug 2, 2021, at 5:20 PM, Bakul Shah wrote: >> >> On Aug 2, 2021, at 2:25 PM, Dan Cross wrote: >>> >>> Fork has served us well for more than five decades; I've got no argument with that. However, should we never question whether it continues to be the right, or best, abstraction as the environment around it continues to evolve? >> >> An os with no fork() can be used to reimplement fork() for backward >> compatibility. You will anyway need all the things fork() used to do >> in this brave new world. > > Oh man, now I’m having flashbacks to early Cygwin and what was that, DJGPP for OS/2? > > Well, I picked a lousy week to give up sniffing glue anyway. > > Adam EMX. -uso. From jon at fourwinds.com Tue Aug 3 11:49:17 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Mon, 02 Aug 2021 18:49:17 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: <202108030149.1731nIsp395841@darkstar.fourwinds.com> Bakul Shah writes: > Or perhaps the issue is not having graphics/GUI designers with the > creativity and sensibilities of the early Bell Labs crowd of researchers? > I keep thinking there ought to be something simpler/more elegant than > the current graphics subsystems.... > > > On Aug 2, 2021, at 11:16 AM, Adam Thornton wrote: > > > > It's a measure of Unix having been wounded by its own success. > > > > fork() is a great model for a single-threaded text processing pipeline to > do automated typesetting. (More generally, anything that is a straightforward > composition of filter/transform stages.) Which is, y'know, what Unix is *for*. > > > > It's not so great for a responsive GUI in front of a multi-function interactive program. > > > > These days, the vast majority of Unix applications are "stuff people play with on their phones." > > > > Adam I thought that I posted something about this recently when someone was arguing for threads being unnecessary and bad. My two cents is that GUIs were the big driver for threads. I would personally like to get rid of them as they're "the same but different" with regards to processes. My preference would be to solve the heaviness of processes problem. I'm not in the thick of that these days, but I don't see it being solved in software alone; it's going to take some serious hardware/software architecture work. Might be easier to accomplish now that the world has pretty much settled on the process model. Jon From athornton at gmail.com Tue Aug 3 13:21:03 2021 From: athornton at gmail.com (Adam Thornton) Date: Mon, 2 Aug 2021 20:21:03 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <202108030149.1731nIsp395841@darkstar.fourwinds.com> References: <202108030149.1731nIsp395841@darkstar.fourwinds.com> Message-ID: <0C986DD0-8399-44DB-B6AD-D966E25975FD@gmail.com> > On Aug 2, 2021, at 6:49 PM, Jon Steinhart wrote: > > My two cents is that GUIs were the big driver for threads. I would > personally like to get rid of them as they're "the same but different" > with regards to processes. My preference would be to solve the > heaviness of processes problem. I'm not in the thick of that these > days, but I don't see it being solved in software alone; it's going > to take some serious hardware/software architecture work. Might be > easier to accomplish now that the world has pretty much settled on > the process model. Has it, though? Most of the stuff I’m working with right now is either asyncio and Python, or Javascript, both of which are very much about threading. I think there’s a lot of stuff that does work better with multiple program counters for different things executing at the same time with shared access to memory, although of course reasoning about concurrency is always hard and having to manage locks on things that shouldn’t be shared _right now_ is a lot of work and easy to get wrong. I like Go and goroutines and channels as an intra-process communication mechanism. My problem with processes, in the traditional sense, is that there ARE often pieces of state you want to share between the things concurrently acting on that state. A channel-like mechanism could work as IPC, but, equally well, why _not_ have multiple things—we can call them threads if you want—that can just see that shared state? Adam From bakul at iitbombay.org Tue Aug 3 13:27:23 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Mon, 2 Aug 2021 20:27:23 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <202108030149.1731nIsp395841@darkstar.fourwinds.com> References: <202108030149.1731nIsp395841@darkstar.fourwinds.com> Message-ID: <55D95A34-3644-4A95-B94F-76EC1A37D729@iitbombay.org> On Aug 2, 2021, at 6:49 PM, Jon Steinhart wrote: > > Bakul Shah writes: >> Or perhaps the issue is not having graphics/GUI designers with the >> creativity and sensibilities of the early Bell Labs crowd of researchers? >> I keep thinking there ought to be something simpler/more elegant than >> the current graphics subsystems.... >> >>> On Aug 2, 2021, at 11:16 AM, Adam Thornton wrote: >>> >>> It's a measure of Unix having been wounded by its own success. >>> >>> fork() is a great model for a single-threaded text processing pipeline to >> do automated typesetting. (More generally, anything that is a straightforward >> composition of filter/transform stages.) Which is, y'know, what Unix is *for*. >>> >>> It's not so great for a responsive GUI in front of a multi-function interactive program. >>> >>> These days, the vast majority of Unix applications are "stuff people play with on their phones." >>> >>> Adam > > I thought that I posted something about this recently when someone > was arguing for threads being unnecessary and bad. > > My two cents is that GUIs were the big driver for threads. I would > personally like to get rid of them as they're "the same but different" > with regards to processes. My preference would be to solve the > heaviness of processes problem. I'm not in the thick of that these > days, but I don't see it being solved in software alone; it's going > to take some serious hardware/software architecture work. Might be > easier to accomplish now that the world has pretty much settled on > the process model. > > Jon AFAIK pretty much all GUI frameworks are (or were) single threaded. At least I don't see the GUI as the main motivation for threads. I was mainly complaining about the complexity of graphics subsystem, including user interaction. Don't see what fork() has to do with it. From jon at fourwinds.com Tue Aug 3 13:51:41 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Mon, 02 Aug 2021 20:51:41 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <55D95A34-3644-4A95-B94F-76EC1A37D729@iitbombay.org> References: <202108030149.1731nIsp395841@darkstar.fourwinds.com> <55D95A34-3644-4A95-B94F-76EC1A37D729@iitbombay.org> Message-ID: <202108030351.1733pfXb494598@darkstar.fourwinds.com> Bakul Shah writes: > On Aug 2, 2021, at 6:49 PM, Jon Steinhart wrote: > > > > Bakul Shah writes: > >> Or perhaps the issue is not having graphics/GUI designers with the > >> creativity and sensibilities of the early Bell Labs crowd of researchers? > >> I keep thinking there ought to be something simpler/more elegant than > >> the current graphics subsystems.... > >> > >>> On Aug 2, 2021, at 11:16 AM, Adam Thornton wrote: > >>> > >>> It's a measure of Unix having been wounded by its own success. > >>> > >>> fork() is a great model for a single-threaded text processing pipeline to > >> do automated typesetting. (More generally, anything that is a straightforward > >> composition of filter/transform stages.) Which is, y'know, what Unix is *for*. > >>> > >>> It's not so great for a responsive GUI in front of a multi-function interactive program. > >>> > >>> These days, the vast majority of Unix applications are "stuff people play with on their phones." > >>> > >>> Adam > > > > I thought that I posted something about this recently when someone > > was arguing for threads being unnecessary and bad. > > > > My two cents is that GUIs were the big driver for threads. I would > > personally like to get rid of them as they're "the same but different" > > with regards to processes. My preference would be to solve the > > heaviness of processes problem. I'm not in the thick of that these > > days, but I don't see it being solved in software alone; it's going > > to take some serious hardware/software architecture work. Might be > > easier to accomplish now that the world has pretty much settled on > > the process model. > > > > Jon > > AFAIK pretty much all GUI frameworks are (or were) single threaded. > At least I don't see the GUI as the main motivation for threads. > > I was mainly complaining about the complexity of graphics subsystem, > including user interaction. Don't see what fork() has to do with it. Not my experience. At least from what I was working on the 1980s there was a need to be able to keep complicated state in graphical programs which is when assembly language versions of threads came into being. Made it possible for whatever was generating output to be interrupted so that input could be handled in a timely manner. I seem to recall getting some code from Sun for this that I ported to a different system. Jon From arnold at skeeve.com Tue Aug 3 17:19:19 2021 From: arnold at skeeve.com (arnold at skeeve.com) Date: Tue, 03 Aug 2021 01:19:19 -0600 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: <202108030719.1737JJRc019869@freefriends.org> "Theodore Ts'o" wrote: > It's certainly clear that some kind of primitive is needed to create > new threads. An open question is whether if there exists some kind of > "new thread" primitve plus either spawn(2) or some kind of "create a > child process and then then frob like crazy using 'echo XXX > > /proc//'" whether there still is a need for a > fork(2) system call. I haven't caught up yet in this thread. Apologies if this has been discussed already. The Plan 9 folks blazed this trail over 30 years ago with rfork, where you specify what bits you wish to duplicate. I don't remember details anymore, but I think it was pretty elegant. IIRC Around that time Rob Pike said "Threads are the lack of an idea", meaning, if you think you need threads, you haven't thought about the problem hard enough. (Apologies to Rob if I am misremembering and/or misrepresenting.) Arnold From douglas.mcilroy at dartmouth.edu Wed Aug 4 01:01:20 2021 From: douglas.mcilroy at dartmouth.edu (Douglas McIlroy) Date: Tue, 3 Aug 2021 11:01:20 -0400 Subject: [TUHS] Systematic approach to command-line interfaces Message-ID: > fork() is a great model for a single-threaded text processing pipeline to do > automated typesetting. (More generally, anything that is a straightforward > composition of filter/transform stages.) Which is, y'know, what Unix is *for*. > It's not so great for a responsive GUI in front of a multi-function interactive program. "Single-threaded" is not a term I would apply to multiple processes in a pipeline. If you mean a single track of data flow, fine, but the fact that that's a prevalent configuration of cooperating processes in Unix is an artifact of shell syntax, not an inherent property of pipe-style IPC. The cooperating processes in Rob Pike's 20th century window systems and screen editors, for example, worked smoothly without interrupts or events - only stream connections. I see no abstract distinction between these programs and "stuff people play with on their phones." It bears repeating, too, that stream connections are much easier to reason about than asynchronous communication. Thus code built on streams is far less vulnerable to timing bugs. At last a prince has come to awaken the sleeping beauty of stream connections. In Go (Pike again) we have a widely accepted programming language that can fully exploit them, "[w]hich is, y'know, what Unix is 'for'." (If you wish, you may read "process" above to include threads, but I'll stay out of that.) Doug From pugs at ieee.org Wed Aug 4 03:13:31 2021 From: pugs at ieee.org (Tom Lyon) Date: Tue, 3 Aug 2021 10:13:31 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: I always believed that fork() was the very essence of the beauty of UNIX. So simple, yet so powerful. And it is far simpler than its predecessor on Genie, which looked a lot more like spawn(). With fork, the child inherits properties which may not have existed when the code was written, so it's much easier to reason about the behavior of sub-processes. Fork made writing the shell and pipelines much more obvious. Today we know that threads and shared mutable memory are a really bad idea, it's just that the hardware gave us no alternatives. I claim UNIX is directly responsible for the existence of MMUs in microprocessors. What if CPU designers would add facilities to directly implement inter-process or inter-processor messaging? Sadly, there has to be a dominant software paradigm for the hardware guys to target, so there's a nasty chicken and egg problem. Imagine if the Erlang model of distributed systems had taken off. Go gets us part of the way there, but cross-machine messaging is still a mess. On Tue, Aug 3, 2021 at 8:02 AM Douglas McIlroy < douglas.mcilroy at dartmouth.edu> wrote: > > fork() is a great model for a single-threaded text processing pipeline > to do > > automated typesetting. (More generally, anything that is a > straightforward > > composition of filter/transform stages.) Which is, y'know, what Unix > is *for*. > > > It's not so great for a responsive GUI in front of a multi-function > interactive program. > > "Single-threaded" is not a term I would apply to multiple processes in > a pipeline. If you mean a single track of data flow, fine, but the > fact that that's a prevalent configuration of cooperating processes in > Unix is an artifact of shell syntax, not an inherent property of > pipe-style IPC. The cooperating processes in Rob Pike's 20th century > window systems and screen editors, for example, worked smoothly > without interrupts or events - only stream connections. I see no > abstract distinction between these programs and "stuff people play > with on their phones." > > It bears repeating, too, that stream connections are much easier to > reason about than asynchronous communication. Thus code built on > streams is far less vulnerable to timing bugs. > > At last a prince has come to awaken the sleeping beauty of stream > connections. In Go (Pike again) we have a widely accepted programming > language that can fully exploit them, "[w]hich is, y'know, what Unix > is 'for'." > > (If you wish, you may read "process" above to include threads, but > I'll stay out of that.) > > Doug > -- - Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From douglas.mcilroy at dartmouth.edu Wed Aug 4 05:12:48 2021 From: douglas.mcilroy at dartmouth.edu (Douglas McIlroy) Date: Tue, 3 Aug 2021 15:12:48 -0400 Subject: [TUHS] Systematic approach to command-line interfaces Message-ID: > Go gets us part of the way there, but cross-machine messaging is still a mess. Shed a tear for Plan 9 (Pike yet again). While many of its secondary innovations have been stuffed into Linux; its animating principle--transparently distributable computing--could not overcome the enormous inertia of installed BSD-model systems. Doug From andreww591 at gmail.com Wed Aug 4 09:12:35 2021 From: andreww591 at gmail.com (Andrew Warkentin) Date: Tue, 3 Aug 2021 17:12:35 -0600 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: <202108030719.1737JJRc019869@freefriends.org> References: <202108030719.1737JJRc019869@freefriends.org> Message-ID: On 8/3/21, arnold at skeeve.com wrote: > > I haven't caught up yet in this thread. Apologies if this has been > discussed already. > > The Plan 9 folks blazed this trail over 30 years ago with rfork, where > you specify what bits you wish to duplicate. I don't remember details > anymore, but I think it was pretty elegant. IIRC Around that time Rob Pike > said "Threads are the lack of an idea", meaning, if you think you need > threads, you haven't thought about the problem hard enough. (Apologies > to Rob if I am misremembering and/or misrepresenting.) > I've never really been a fan of the rfork()/clone() model, or at least the Linux implementation of it that requires ugly library-level hacks to share state between threads that the kernel doesn't support sharing. Also, I don't really care for the large number of flags required. Up until now I was just planning on following the traditional threading model of associating most state with processes with only execution state being per-thread in the OS I'm working on, but now I'm thinking I should reduce the state associated with a process to just the PID, PPID, PGID, containing cgroup, command line, and list of threads. All other state would be contained in various types of context objects that are not tied to a particular process or thread (other than being destroyed when no more threads are associated with them). This would include: Filesystem namespace File descriptors Address space Security context (file permission list, UID, GID) Signal handlers Scheduling context Each of these object types would be completely separate from the others, allowing full control over which state is shared and which is private. I'm using seL4 as a microkernel, and it already works like this (it has no real concept of processes, only threads that are each associated with an address space, a capability space, and a scheduling context) so it's a good match for it. exec() would still replace all threads within a process as on traditional Unix, unless the exec is performed within a child process that hasn't yet been started. Sending a signal to an entire process would send it to every signal group within the process (similarly, it would be possible to send a signal to an entire cgroup; basically, processes will really just be a special kind of cgroup in this model). From g.branden.robinson at gmail.com Wed Aug 4 18:16:53 2021 From: g.branden.robinson at gmail.com (G. Branden Robinson) Date: Wed, 4 Aug 2021 18:16:53 +1000 Subject: [TUHS] Why does refer(1) have no database field for "edition"?. In-Reply-To: References: Message-ID: <20210804081652.pr27a53zc7tnpkva@localhost.localdomain> So many groff discussions get re-routed to TUHS thanks to Doug's MUA. :) At 2021-08-02T19:38:42-0400, Douglas McIlroy wrote: > Another issue I have run into is recursion, when a reference includes > another reference. This comes up in various forms: > Also published as ... > Errata available at ... > Summarized [or reviewed] in ... > Preprint available at ... > Often such a reference takes the form of a URL or a page in a journal > or in a proceedings. This can be most succinctly placed in situ -- > formatted consistently with other references. If the reference > identifies an alternate mode of publication, it may lack %A or %T > fields. > > Partial proposal. a %O field may contain a reference, with no further > recursion, The contained reference will be formatted in line in the %O > text unless references are accumulated and the contained reference is > not unique. This would tighten the existing semantics of %O, would it not? Maybe that's not a problem, since it's already defined as a dumping ground[1]. So if I understand you and the GNU refer(1) manual correctly, %O would be parsed for a pair of brackets[2] containing keywords only (no fields), and if that resolved to a single reference, the label for that reference would be interpolated. That of itself would be sufficient to halt the recursion. Here's an example of what this sort of thing looks like today, _without_ adding this feature to %O[3]. $ cat EXPERIMENTS/my-db-file-for-doug %A Daniel P.\& Friedman %A Matthias Felleisen %C Cambridge, Massachusetts %D 1996 %I The MIT Press %T The Little Schemer, Fourth Edition %O Reviewed in [posterior analytics 36] $ REFER=EXPERIMENTS/my-db-file-for-doug ./build/test-groff -R -Tutf8 \ -ms EXPERIMENTS/scheme.ms | cat -s Read the book[1] on your summer vacation. Commentary is available.* ─────────── [1] Daniel P. Friedman and Matthias Felleisen, The Little Schemer, Fourth Edition, The MIT Press, Cam‐ bridge, Massachusetts (1996). Reviewed in [posterior analytics 36]. * Space reserved for penetrating insight. Regards, Branden [1] "Other information. This is usually printed at the end of the reference." [2] could be anything non-keywordy, but brackets seem like a good choice [3] ...and a nickel goes into the "cat -v swear jar". -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From paul.winalski at gmail.com Thu Aug 5 01:04:50 2021 From: paul.winalski at gmail.com (Paul Winalski) Date: Wed, 4 Aug 2021 11:04:50 -0400 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: <202108030719.1737JJRc019869@freefriends.org> Message-ID: On 8/3/21, Andrew Warkentin wrote: > > Up until now I was just planning on following the traditional > threading model of associating most state with processes with only > execution state being per-thread in the OS I'm working on, but now I'm > thinking I should reduce the state associated with a process to just > the PID, PPID, PGID, containing cgroup, command line, and list of > threads. All other state would be contained in various types of > context objects that are not tied to a particular process or thread > (other than being destroyed when no more threads are associated with > them). For what it's worth, this is the process model that Windows NT has. This thread has discussed two very different ways to do command line processing: the TOPS-20 model, where the program image for the command is activated immediately and then calls back to the OS to process the command line elements, and the UNIX model, where the first element of the command line is a path (full or partial) to the program image for the command, and it's entirely up to that image to process the command line elements as it sees fit. VMS takes a third approach. The VAX had four, hierarchical execution modes: user, supervisor, exec, and kernel, in increasing order of privilege. The VAX had "change mode" instructions (CHMU, CHMS, CHME, CHMK) that ran the current process's change-mode interrupt handler to process the request. Applications and programs to process commands ran in user mode. The DCL command interpreter (VMS's equivalent of the UNIX shell) ran in supervisor mode. Exec mode was reserved for record management services (RMS), VMS's equivalent of the file system code in UNIX. Kernel mode (the only mode where privileged instructions can be executed) was, of course, for the kernel. One oddity of VMS is that each process had the supervisor, exec, and kernel code and data mapped to its address space. Commands in VMS are described using a meta-language that is compiled into a set of data tables that the command line interpreter (CLI, running in supervisor mode) uses to determine which program image needs to be activated to execute the command. The CLI also does a full parse of the command line. It then calls the kernel's image activator to map the appropriate program image and switches to user mode to start it running. There is a runtime library of routines to retrieve the various options and parameters from the command line (the routines in this library do CHMS calls to achieve this). So on VMS we have a situation where the command interpreter (shell) is part of every process. Processes are created by the "create process" OS library routine, sys$creprc, which is a wrapper around the appropriate CHMK instruction. Unlike fork() on UNIX, the new process inherits very little state from its parent. Most significantly it inherits neither address space nor open file context. VMS process creation is also notoriously expensive compared to fork() on UNIX. -Paul W. From stewart at serissa.com Thu Aug 5 07:48:46 2021 From: stewart at serissa.com (Lawrence Stewart) Date: Wed, 4 Aug 2021 17:48:46 -0400 Subject: [TUHS] Threads vs... not Message-ID: What do folks think about event-driven programming as a substitute for threads in UI and process control settings? I wrote the service processor code for the Sicortex Machines using libevent.a and I thought it was very lightweight and fairly easy to think about. (This was a thing running on ucLinux on a tiny 16 MB coldfire that managed the consoles and power supplies and temp sensors and JTAG access and booting and so forth.) Tk (IIRC) has a straightforward event driven model for UI interactions. Meanwhile, the dropbox plugin for my Mac has 120 threads running. WTF? This was triggered by the fork/spawn discussion. -Larry (started with Unix at V6 on an 11/34) From lm at mcvoy.com Thu Aug 5 08:02:08 2021 From: lm at mcvoy.com (Larry McVoy) Date: Wed, 4 Aug 2021 15:02:08 -0700 Subject: [TUHS] Threads vs... not In-Reply-To: References: Message-ID: <20210804220208.GK9074@mcvoy.com> >From http://mcvoy.com/lm/quotes.html A computer is a state machine. Threads are for people who can't program state machines. Alan Cox On Wed, Aug 04, 2021 at 05:48:46PM -0400, Lawrence Stewart wrote: > What do folks think about event-driven programming as a substitute for threads in UI and process control settings? > > I wrote the service processor code for the Sicortex Machines using libevent.a and I thought it was very lightweight and fairly easy to think about. (This was a thing running on ucLinux on a tiny 16 MB coldfire that managed the consoles and power supplies and temp sensors and JTAG access and booting and so forth.) > > Tk (IIRC) has a straightforward event driven model for UI interactions. > > Meanwhile, the dropbox plugin for my Mac has 120 threads running. WTF? > > This was triggered by the fork/spawn discussion. > > -Larry > (started with Unix at V6 on an 11/34) -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From ality at pbrane.org Thu Aug 5 08:13:37 2021 From: ality at pbrane.org (Anthony Martin) Date: Wed, 4 Aug 2021 15:13:37 -0700 Subject: [TUHS] Threads vs... not In-Reply-To: References: Message-ID: Lawrence Stewart once said: > What do folks think about event-driven programming as a substitute for > threads in UI and process control settings? Why not both? https://swtch.com/~rsc/talks/threads07/ https://swtch.com/~rsc/thread/cws.pdf Cheers, Anthony From cowan at ccil.org Thu Aug 5 08:41:08 2021 From: cowan at ccil.org (John Cowan) Date: Wed, 4 Aug 2021 18:41:08 -0400 Subject: [TUHS] Threads vs... not In-Reply-To: <20210804220208.GK9074@mcvoy.com> References: <20210804220208.GK9074@mcvoy.com> Message-ID: On Wed, Aug 4, 2021 at 6:02 PM Larry McVoy wrote: > A computer is a state machine. Threads are for people who can't > program > state machines. > > Alan Cox Orly? Try embedding an LL(1) parser in an event loop that gives you a new event every time a block is read off the disk. Event loops are just manual CPS transformations of coroutines -- but why do the transformation manually instead of having your compiler do it for you? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lm at mcvoy.com Thu Aug 5 08:51:07 2021 From: lm at mcvoy.com (Larry McVoy) Date: Wed, 4 Aug 2021 15:51:07 -0700 Subject: [TUHS] Threads vs... not In-Reply-To: References: <20210804220208.GK9074@mcvoy.com> Message-ID: <20210804225107.GL9074@mcvoy.com> On Wed, Aug 04, 2021 at 06:41:08PM -0400, John Cowan wrote: > On Wed, Aug 4, 2021 at 6:02 PM Larry McVoy wrote: > > > > A computer is a state machine. Threads are for people who can't > > program > > state machines. > > > > Alan Cox > > > Orly? Try embedding an LL(1) parser in an event loop that gives you a new > event every time a block is read off the disk. > > Event loops are just manual CPS transformations of coroutines -- but why do > the transformation manually instead of having your compiler do it for you? The counter to this is Solaris tried to allocate a thread for each 8K page on it's way to disk. The thread stack was 16K. This model, while seen as ever so elegant, means that 2/3rds of main memory were thread stacks. Sometimes threads make sense, there they did not. -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm From john at jfloren.net Thu Aug 5 10:51:11 2021 From: john at jfloren.net (John Floren) Date: Thu, 05 Aug 2021 00:51:11 +0000 Subject: [TUHS] Depraz/Logitech Digimouse manual Message-ID: Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. Thanks, john -------------- next part -------------- An HTML attachment was scrubbed... URL: From henry.r.bent at gmail.com Thu Aug 5 11:12:47 2021 From: henry.r.bent at gmail.com (Henry Bent) Date: Wed, 4 Aug 2021 21:12:47 -0400 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: Message-ID: On Wed, 4 Aug 2021 at 20:52, John Floren wrote: > Having just been given a Depraz mouse, I thought it would be fun to get it > working on my modern computer. Since the DE9 connector is male rather than > female as you usually see with serial mice, and given its age, I speculate > that it might have a custom protocol; in any rate, plugging it into a > USB-serial converter and and firing up picocom has given me nothing. > > Does anyone have a copy of a manual for it, or more information on how to > interface with it? If I knew how it was wired and what the protocol looked > like, I expect I could make an adapter pretty trivially using a > microcontroller. > This might be of some help? https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 -Henry -------------- next part -------------- An HTML attachment was scrubbed... URL: From bakul at iitbombay.org Thu Aug 5 12:10:30 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Wed, 4 Aug 2021 19:10:30 -0700 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: Message-ID: <18809DBF-3FBF-47C2-AD36-D4A33D6A36B4@iitbombay.org> On Aug 4, 2021, at 5:51 PM, John Floren wrote: > > Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. > > Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. You'll likely need a couple of quadrature decoders as X1,X2 (& Y1,Y2) transitions encode the direction of movement in X (& Y) axis. This may help: http://www.mcmanis.com/chuck/robotics/projects/lab-x3/quadratrak.html On a 'Pi may be you can just wire it up to a few GPIO pins (but you will need to convert 5V signals to 3.3V to avoid damaging them). -- Bakul From john at jfloren.net Thu Aug 5 13:58:41 2021 From: john at jfloren.net (John Floren) Date: Thu, 05 Aug 2021 03:58:41 +0000 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: Message-ID: ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent wrote: > On Wed, 4 Aug 2021 at 20:52, John Floren wrote: > > > Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. > > > > Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. > > This might be of some help? > > https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 > > -Henry This looks great, thank you! john From rminnich at gmail.com Thu Aug 5 14:12:04 2021 From: rminnich at gmail.com (ron minnich) Date: Wed, 4 Aug 2021 21:12:04 -0700 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: Message-ID: John, you can see that "stick a bird on it" -> "stick an arduino on it" -> "stick a pi on it" has gone as you once predicted :-) On Wed, Aug 4, 2021 at 8:59 PM John Floren wrote: > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent wrote: > > > On Wed, 4 Aug 2021 at 20:52, John Floren wrote: > > > > > Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. > > > > > > Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. > > > > This might be of some help? > > > > https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 > > > > -Henry > > This looks great, thank you! > > john From bakul at iitbombay.org Thu Aug 5 14:14:37 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Wed, 4 Aug 2021 21:14:37 -0700 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: Message-ID: The 'Pi that is running Plan9 or some Unix, not a separate one! -- Bakul > On Aug 4, 2021, at 9:12 PM, ron minnich wrote: > > John, you can see that "stick a bird on it" -> "stick an arduino on > it" -> "stick a pi on it" has gone as you once predicted :-) > > On Wed, Aug 4, 2021 at 8:59 PM John Floren wrote: >> >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent wrote: >> >>> On Wed, 4 Aug 2021 at 20:52, John Floren wrote: >>> >>>> Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. >>>> >>>> Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. >>> >>> This might be of some help? >>> >>> https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 >>> >>> -Henry >> >> This looks great, thank you! >> >> john From clemc at ccc.com Fri Aug 6 06:32:19 2021 From: clemc at ccc.com (Clem Cole) Date: Thu, 5 Aug 2021 16:32:19 -0400 Subject: [TUHS] Threads vs... not In-Reply-To: <20210804225107.GL9074@mcvoy.com> References: <20210804220208.GK9074@mcvoy.com> <20210804225107.GL9074@mcvoy.com> Message-ID: On Wed, Aug 4, 2021 at 6:51 PM Larry McVoy wrote: > Sometimes threads make sense, there they did not. Right. The problem is too often people have a hammer and make everything a nail. Or they have an electric screwdriver and then try to use if for everything. The trick is to teach good taste and when certain solutions make more sense than others. Allocating a thread for every page being sent to disk, or needing 120 threads for a simple application -- somebody missed the boat. Funny, used to deliver nails in kegs. Hit with a hammer - wood fastened. Somebody develops a screw system. A 'better fastener.' Indeed it does work great for >>some jobs<< but not all. Frankly that sucks for framing. What do you see at any real job site. Not screws but a pneumatic hammer and nail cartridges? Changed the deliver system from key and forcing somebody to swing the hammer. Lesson learned to *use the proper tool *or update the tool if the >>technique<< is the right one. Kids these days ... get off my lawn... Clem -------------- next part -------------- An HTML attachment was scrubbed... URL: From bakul at iitbombay.org Fri Aug 6 09:02:06 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Thu, 5 Aug 2021 16:02:06 -0700 Subject: [TUHS] Threads vs... not In-Reply-To: References: Message-ID: <74E8349E-95A4-40C9-B429-11A4E396BE12@iitbombay.org> On Aug 4, 2021, at 2:48 PM, Lawrence Stewart wrote: > > What do folks think about event-driven programming as a substitute for threads in UI and process control settings? The equivalence here is with coroutines (sometimes called green-threads), not true threads. True threads can take advantage of multiple cores. Your event-driven program won't benefit from multiple cores. > I wrote the service processor code for the Sicortex Machines using libevent.a and I thought it was very lightweight and fairly easy to think about. (This was a thing running on ucLinux on a tiny 16 MB coldfire that managed the consoles and power supplies and temp sensors and JTAG access and booting and so forth.) I wrote the RealNetworks media server code using an event-driven model. It was efficient but the control flow got quite messy as effectively my code had to do explicit continuation passing. Given that each media stream was independent of other streams, a thread based model was far simpler. That is what a colleague did to test the server code! As a thought experiment consider implementing a Unix style pipeline but without an OS that provides the illusion of independent processes or threads and where each stage in the pipeline is a separate function. Try implementing this in Go which has support for threads vs C (but without pthreads). I have used coroutines for hardware simulation a number of times, the first time was in 1982 for simulating bus transactions to see the impact of adding an ethernet controller (5.6Mhz clock, memory access 4 clocks). The last time was to simulate an ethernet switch with complex i/o logic. One can think of each hardware module as a separate state machine but smushing them into one giant state machine would not been easy as the code was being developed and debugging it would be very painful. One would then be tempted to write a "preprocessor" language of some sort to hide all this.... > Tk (IIRC) has a straightforward event driven model for UI interactions. > > Meanwhile, the dropbox plugin for my Mac has 120 threads running. WTF? Any system can be abused but the abuse-case should not be mistaken for the normal use-case. From crossd at gmail.com Fri Aug 6 10:55:18 2021 From: crossd at gmail.com (Dan Cross) Date: Thu, 5 Aug 2021 20:55:18 -0400 Subject: [TUHS] Threads vs... not In-Reply-To: <20210804225107.GL9074@mcvoy.com> References: <20210804220208.GK9074@mcvoy.com> <20210804225107.GL9074@mcvoy.com> Message-ID: On Wed, Aug 4, 2021 at 6:51 PM Larry McVoy wrote: > On Wed, Aug 04, 2021 at 06:41:08PM -0400, John Cowan wrote: > > On Wed, Aug 4, 2021 at 6:02 PM Larry McVoy wrote: > > > A computer is a state machine. Threads are for people who can't > > > program state machines. > > > > > > Alan Cox > > > > Orly? Try embedding an LL(1) parser in an event loop that gives you a > new > > event every time a block is read off the disk. > > > > Event loops are just manual CPS transformations of coroutines -- but why > do > > the transformation manually instead of having your compiler do it for > you? > > The counter to this is Solaris tried to allocate a thread for each 8K page > on it's way to disk. The thread stack was 16K. This model, while seen as > ever so elegant, means that 2/3rds of main memory were thread stacks. > Honestly this sounds like a misapplication of threads. It makes more sense to me to have a small pool of workers working on a bounded queue or something. Sometimes threads make sense, there they did not. It strikes me that the Unix kernel we all know and love has been multithreaded since the early 1970s, when multiprogramming overlapped with IO was introduced; even in something as simple as 6th Edition, every process has a corresponding kernel thread and the fundamental mechanism to context switch from process $n$ to process $m$ is for n to trap into the kernel where it's running on n's kernel thread, invoke the scheduler to perform a thread switch to m's kernel thread, and then return to m's userspace. Various attempts at replacing multithreaded kernels with event-driven state machines have been proposed and tried, but in the end, I always come away with the sense that most of the time, when used judiciously, threads are wonderful concurrency primitives. - Dan C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steffen at sdaoden.eu Sat Aug 7 00:19:24 2021 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Fri, 06 Aug 2021 16:19:24 +0200 Subject: [TUHS] Threads vs... not In-Reply-To: <74E8349E-95A4-40C9-B429-11A4E396BE12@iitbombay.org> References: <74E8349E-95A4-40C9-B429-11A4E396BE12@iitbombay.org> Message-ID: <20210806141924.O2Y1R%steffen@sdaoden.eu> Bakul Shah wrote in <74E8349E-95A4-40C9-B429-11A4E396BE12 at iitbombay.org>: |On Aug 4, 2021, at 2:48 PM, Lawrence Stewart wrote: |> |> What do folks think about event-driven programming as a substitute \ |> for threads in UI and process control settings? | |The equivalence here is with coroutines (sometimes called |green-threads), not true threads. True threads can take |advantage of multiple cores. Your event-driven program won't |benefit from multiple cores. | |> I wrote the service processor code for the Sicortex Machines using \ |> libevent.a and I thought it was very lightweight and fairly easy \ |> to think about. (This was a thing running on ucLinux on a tiny 16 \ |> MB coldfire that managed the consoles and power supplies and temp \ |> sensors and JTAG access and booting and so forth.) | |I wrote the RealNetworks media server code using an |event-driven model. It was efficient but the control flow got |quite messy as effectively my code had to do explicit |continuation passing. Given that each media stream was |independent of other streams, a thread based model was far |simpler. That is what a colleague did to test the server |code! Only twenty years ago but i was under the impression that i got good (better) performance by having a single event loop object (thread) doing the select(2) aka the OS interaction, as the driver under the hood of an IOEvent thing, which then were emitted. These then dispatched to worker threads, or not. It may be different today with those highly refined mechanisms poll, epoll, kqueue. It may anyway be different with whatever operating system specific things, for example FreeBSD had or has some network ring where user and kernel space could simply exchange buffers, ?, well i remember it was written by an Italien (Professor?) living in Pisa, in sight of the lopsided tower. And then the audioring technology Creative Labs shipped with X-Fi ~2005. --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From cowan at ccil.org Sat Aug 7 00:43:17 2021 From: cowan at ccil.org (John Cowan) Date: Fri, 6 Aug 2021 10:43:17 -0400 Subject: [TUHS] Threads vs... not In-Reply-To: <20210806141924.O2Y1R%steffen@sdaoden.eu> References: <74E8349E-95A4-40C9-B429-11A4E396BE12@iitbombay.org> <20210806141924.O2Y1R%steffen@sdaoden.eu> Message-ID: On Fri, Aug 6, 2021 at 10:20 AM Steffen Nurpmeso wrote: > Bakul Shah wrote in > <74E8349E-95A4-40C9-B429-11A4E396BE12 at iitbombay.org>: > |It was efficient but the control flow got > |quite messy as effectively my code had to do explicit > |continuation passing. > > Only twenty years ago but i was under the impression that i got > good (better) performance by having a single event loop object > (thread) doing the select(2) aka the OS interaction, as the driver > under the hood of an IOEvent thing, which then were emitted. > It sounds like you are in violent agreement: you get more performance with an event loop. So perhaps Cox's claim is true after all: threads are for programmers who can't deal with events. And then my claim is that at a certain not-very-high level of complexity no one can deal with events. After all, the interface to all modern operating systems is as a coroutine: when we call read() we give up control to the kernel, who does the read for us and delivers it to a buffer in our space, then surrenders control back. (How it looks from inside the kernel is another story.) Only when we request signal handling does the kernel actually interrupt the ordered flow within our program, but the restrictions on what a signal handler can do in Posix are so many that it's hardly safe to do more than set a flag to be polled later or to do a longjmp() to invoke a continuation further up the stack. Historic operating systems worked quite differently: they had the equivalent of signals that meant "disk block has been read" (use the buffer, and fast, before the OS starts to write the next block into it) and "disk block has been written" (the buffer is available for reuse). And programming for them was, in someone's memorable phrase, like moving dead whales along the beach by kicking them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steffen at sdaoden.eu Sat Aug 7 01:12:23 2021 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Fri, 06 Aug 2021 17:12:23 +0200 Subject: [TUHS] Threads vs... not In-Reply-To: References: <74E8349E-95A4-40C9-B429-11A4E396BE12@iitbombay.org> <20210806141924.O2Y1R%steffen@sdaoden.eu> Message-ID: <20210806151223._GD-D%steffen@sdaoden.eu> John Cowan wrote in : |On Fri, Aug 6, 2021 at 10:20 AM Steffen Nurpmeso \ |wrote: |> Bakul Shah wrote in |> <74E8349E-95A4-40C9-B429-11A4E396BE12 at iitbombay.org>: |>|It was efficient but the control flow got |>|quite messy as effectively my code had to do explicit |>|continuation passing. |> |> Only twenty years ago but i was under the impression that i got |> good (better) performance by having a single event loop object |> (thread) doing the select(2) aka the OS interaction, as the driver |> under the hood of an IOEvent thing, which then were emitted. |> | |It sounds like you are in violent agreement: you get more performance with |an event loop. So perhaps Cox's claim is true after all: threads are for |programmers who can't deal with events. And then my claim is that at a |certain not-very-high level of complexity no one can deal with events. Well i am just too old and lazy to really dig into the asynchronous hype you see "everywhere". I know just for fun i did some MacOS Audio thing around 2009/10 (when i bought the single Apple i once wanted), nothing but a simple OggVorbis player (because Quicktime just did not like free Audio, and because i was interested), and those callbacks i had to deal with just drove me insane. The Linux Jack Audio Server also does that, i think. I do not understand. The CPU usage was just _grazy_, even though i used the biggest buffers i could use (iirc). I mean, i could drive audio play and recording side-by-side (with programming work) on a Cyrix 166+ with a 3 MB/sec (30 MB/sec with caching, there was some I/O test on Linux, forgotten the name) hard disk on FreeBSD 4.7 or what, it was nice(1)d, but other than that, and i think under 1 or 3 percent CPU. That is how it has to be! And not to forget most async and maybe even many coroutine things just internally base on threads, i think OpenSSL for example does, so what looks so cheap and "tickless" is in effect a tremendous monster. I hope they at least use pools. And that the pools do not leak resources to wherever. No, by chance, i am simple-minded and drove non-blocking event loops, which nicely bundled I/O events and gave me timers for free. This scaled wonderful. And for multi-core and super efficiency it scales easily, too: either by having worker threads (or sandboxed, isolated processes), picked out of a pool, doing work as it happens, or maybe, today, scaled by X different event loops. (Like, for networking, having one accepting new connections, and several others handling them.) But this seems so "hyper", i wonder whether any of the actually used "super servers", say postfix, dovecot, exim, postfix, *sql, apache, nginx, let me add lighttpd now that i said postfix, as it works so nice, uses _so_ much scaling, in the actual everyday work. --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From john at jfloren.net Sat Aug 7 02:27:25 2021 From: john at jfloren.net (John Floren) Date: Fri, 06 Aug 2021 16:27:25 +0000 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: Message-ID: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> I stuck an Arduino on it and with surprisingly little code I have it acting like a 3-button USB mouse. The only problem is that the pointer doesn't move smoothly. It does OK left-to-right, and can move down pretty well, but going up is a problem. I think pushing the mouse forward tends to move the ball away from the Y-axis wheel, and the old spring on the tensioner just doesn't have the gumption to hold that heavy ball bearing in any more. john ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, August 4th, 2021 at 9:12 PM, ron minnich wrote: > John, you can see that "stick a bird on it" -> "stick an arduino on > > it" -> "stick a pi on it" has gone as you once predicted :-) > > On Wed, Aug 4, 2021 at 8:59 PM John Floren john at jfloren.net wrote: > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent henry.r.bent at gmail.com wrote: > > > > > On Wed, 4 Aug 2021 at 20:52, John Floren john at jfloren.net wrote: > > > > > > > Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. > > > > > > > > Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. > > > > > > This might be of some help? > > > > > > https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 > > > > > > -Henry > > > > This looks great, thank you! > > > > john From rminnich at gmail.com Sat Aug 7 02:34:25 2021 From: rminnich at gmail.com (ron minnich) Date: Fri, 6 Aug 2021 09:34:25 -0700 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> Message-ID: john, don't forget to mention the beer can On Fri, Aug 6, 2021 at 9:29 AM John Floren wrote: > > I stuck an Arduino on it and with surprisingly little code I have it acting like a 3-button USB mouse. > > The only problem is that the pointer doesn't move smoothly. It does OK left-to-right, and can move down pretty well, but going up is a problem. I think pushing the mouse forward tends to move the ball away from the Y-axis wheel, and the old spring on the tensioner just doesn't have the gumption to hold that heavy ball bearing in any more. > > > john > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > On Wednesday, August 4th, 2021 at 9:12 PM, ron minnich wrote: > > > John, you can see that "stick a bird on it" -> "stick an arduino on > > > > it" -> "stick a pi on it" has gone as you once predicted :-) > > > > On Wed, Aug 4, 2021 at 8:59 PM John Floren john at jfloren.net wrote: > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent henry.r.bent at gmail.com wrote: > > > > > > > On Wed, 4 Aug 2021 at 20:52, John Floren john at jfloren.net wrote: > > > > > > > > > Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. > > > > > > > > > > Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. > > > > > > > > This might be of some help? > > > > > > > > https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 > > > > > > > > -Henry > > > > > > This looks great, thank you! > > > > > > john From john at jfloren.net Sat Aug 7 02:52:15 2021 From: john at jfloren.net (John Floren) Date: Fri, 06 Aug 2021 16:52:15 +0000 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> Message-ID: Ah, right. I opened the mouse because one of the encoders didn't seem to be working (it worked fine again this morning, who knows...) and discovered that there was something duct taped inside the plastic shell: http://jfloren.net/content/depraz/inside.jpg Peeling back the tape, I saw what I first took to be chunks of flattened beer cans: http://jfloren.net/content/depraz/reveal.jpg A closer look showed that they were the wrappers which cover the corks of wine bottles. Up into the 1980s, these were made out of lead, and by flattening five of them, a previous owner of the mouse was able to add quite a bit of extra weight to it: http://jfloren.net/content/depraz/wrapper.jpg john ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, August 6th, 2021 at 9:34 AM, ron minnich wrote: > john, don't forget to mention the beer can > > On Fri, Aug 6, 2021 at 9:29 AM John Floren john at jfloren.net wrote: > > > I stuck an Arduino on it and with surprisingly little code I have it acting like a 3-button USB mouse. > > > > The only problem is that the pointer doesn't move smoothly. It does OK left-to-right, and can move down pretty well, but going up is a problem. I think pushing the mouse forward tends to move the ball away from the Y-axis wheel, and the old spring on the tensioner just doesn't have the gumption to hold that heavy ball bearing in any more. > > > > john > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > On Wednesday, August 4th, 2021 at 9:12 PM, ron minnich rminnich at gmail.com wrote: > > > > > John, you can see that "stick a bird on it" -> "stick an arduino on > > > > > > it" -> "stick a pi on it" has gone as you once predicted :-) > > > > > > On Wed, Aug 4, 2021 at 8:59 PM John Floren john at jfloren.net wrote: > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > > > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent henry.r.bent at gmail.com wrote: > > > > > > > > > On Wed, 4 Aug 2021 at 20:52, John Floren john at jfloren.net wrote: > > > > > > > > > > > Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. > > > > > > > > > > > > Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. > > > > > > > > > > This might be of some help? > > > > > > > > > > https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 > > > > > > > > > > -Henry > > > > > > > > This looks great, thank you! > > > > > > > > john From rminnich at gmail.com Sat Aug 7 04:33:14 2021 From: rminnich at gmail.com (ron minnich) Date: Fri, 6 Aug 2021 11:33:14 -0700 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> Message-ID: The story of the mice, one of which I gave to John: I ran a program called FAST-OS for LANL/Sandia for 6 years starting 2005. Think of it as "Plan 9 on petaflop supercomputers" -- it may seem strange now, but in that era when some top end systems ran custom kernels, there was a strong case to be made that plan 9 was a good choice. By 2011, of course, the Linux tsunami had swept all before it, which is why you no longer hear about custom HPC kernels so much -- though in some places they still reign. In any event, this program gave me 6 years to work with "the Unix room", or what was left of it. I had been in the Unix Room in 1978, and even met Dennis, so this prospect was quite a treat. We funded Charles Forsyth to write the amd64 compilers for Plan 9, which if you used early Go you ran into (6c 6a 6l); we also funded the amd64 port of Plan 9 (a.k.a. k10) as well as the port to Blue Gene. That amd64 port is still out and about. You can find the Blue Gene kernel on github. I had lots of fun spending time in the Unix room while working with the late Jim McKie, and others. I saw the tail end of the traditions. They had cookie day once a week, if memory serves, on Thursday at 3. I got to see the backwards-running clock, Ken's chess trophies, his pilot's license, pictures of Peter everywhere, a "Reagan's view of the world" map, the American Legion award for Telstar (which was rescued from a dumpster!), and so on. The "Unix room" was more than one room, all built on a raised floor, as I assume it was former old school machine room space. If memory serves, it filled the entire width of the end of the top floor of the building it was in (4th floor?) -- maybe 50 ft x 50 ft -- maybe a bit more. There was a room with desks, and a similar-sized room with servers, and a smaller room containing a lab-style sink, a very professional cappucinno machine, decades of old proceedings, and a sofa. I fixed the heavy-duty coffee grinder one year; for some reason the Italian company that produced it had seen fit to switch BOTH hot and neutral, and the fix was to only switch hot, as the neutral switch had failed; I guess in the EU, with 220v, things are done differently. It was fun being there. A few years later the whole room, and all its history, was trashed, and replaced with what Jim called a "middle management wxx dream" (Jim was never at a loss for words); Jim found some yellow Police crime scene tape and placed it in front of the doors to the new space. It was redubbed "the innovation space" or some such, and looked kind of like an ikea showroom. Much was lost. I tried to find a way to save the contents of the room; I had this dream of recreating it at Google, much as John Wanamaker's office was preserved in Philadelphia for so many decades, but I was too late. I have no idea where the contents are now. Maybe next to the Ark. One day in 2008 or so jmk took me for a tour of the buildings, and we at one point ended up high in the top floor of what I think was Building One (since torn down?), in what used to be Lab Supply. Nobody was there, and not much supply was there either. Finally somebody wandered in, and Jim asked where everyone was. "Oh, they closed lab supply, maybe 4 years ago?" Bell Labs had seen hard times since the Lucent split, and it was clear it had not quite recovered, and Lab Supply was just one sign of it. I think the saddest thing was seeing the visitor center, which I first saw in 1976. In 1976, it was the seat of the Bell System Empire, and it was huge. There was a map of the US with a light lit for every switching office in the Bell Labs system. There was all kinds of Bell Labs history in the visitor center museum. The museum had shrunk to a much smaller area, and felt like a closet. The original transistor was still there in 2010, but little else.The library was, similarly, changed: it was dark and empty, I was told. Money was saved. At that time, Bell Labs felt large, strangely quiet, and emptied of people. It made me think of post-sack Rome, ca. 600, when its population was estimated to be 500. I have not been back since 2011 so maybe things are very different. It would be nice if so. As part of this tour, Jim gave me 3 depraz mice. I took one, gutted it, (sorry!), and filled its guts with a USB mouse innards, and gave it back to Jim. He then had a Depraz USB mouse. jmk's mouse did not have any lead in it, as John's did, however. The second I gave to someone at Google who had worked at the labs back in the day. The third mouse I gave to John, and he made it live again, which is cool. In spite of their reputation, I found Depraz mice hard to use. I have gone through all kinds of mice, and am on an evoluent, and as far as Depraz go, I guess "you had to be there". I don't recall if jmk used his "usb depraz" or it ended up on a shelf. Sadly, I can no longer ask him. I'll be interested to see what John thinks of the Depraz. ron On Fri, Aug 6, 2021 at 9:52 AM John Floren wrote: > > Ah, right. I opened the mouse because one of the encoders didn't seem to be working (it worked fine again this morning, who knows...) and discovered that there was something duct taped inside the plastic shell: > > http://jfloren.net/content/depraz/inside.jpg > > Peeling back the tape, I saw what I first took to be chunks of flattened beer cans: > > http://jfloren.net/content/depraz/reveal.jpg > > A closer look showed that they were the wrappers which cover the corks of wine bottles. Up into the 1980s, these were made out of lead, and by flattening five of them, a previous owner of the mouse was able to add quite a bit of extra weight to it: > > http://jfloren.net/content/depraz/wrapper.jpg > > > john > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > On Friday, August 6th, 2021 at 9:34 AM, ron minnich wrote: > > > john, don't forget to mention the beer can > > > > On Fri, Aug 6, 2021 at 9:29 AM John Floren john at jfloren.net wrote: > > > > > I stuck an Arduino on it and with surprisingly little code I have it acting like a 3-button USB mouse. > > > > > > The only problem is that the pointer doesn't move smoothly. It does OK left-to-right, and can move down pretty well, but going up is a problem. I think pushing the mouse forward tends to move the ball away from the Y-axis wheel, and the old spring on the tensioner just doesn't have the gumption to hold that heavy ball bearing in any more. > > > > > > john > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > On Wednesday, August 4th, 2021 at 9:12 PM, ron minnich rminnich at gmail.com wrote: > > > > > > > John, you can see that "stick a bird on it" -> "stick an arduino on > > > > > > > > it" -> "stick a pi on it" has gone as you once predicted :-) > > > > > > > > On Wed, Aug 4, 2021 at 8:59 PM John Floren john at jfloren.net wrote: > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > > > > > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent henry.r.bent at gmail.com wrote: > > > > > > > > > > > On Wed, 4 Aug 2021 at 20:52, John Floren john at jfloren.net wrote: > > > > > > > > > > > > > Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. > > > > > > > > > > > > > > Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. > > > > > > > > > > > > This might be of some help? > > > > > > > > > > > > https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 > > > > > > > > > > > > -Henry > > > > > > > > > > This looks great, thank you! > > > > > > > > > > john From andrew at humeweb.com Sat Aug 7 05:16:46 2021 From: andrew at humeweb.com (Andrew Hume) Date: Fri, 6 Aug 2021 12:16:46 -0700 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> Message-ID: <3C6BAD72-7A29-427C-BDE2-AD6636E5EAD7@humeweb.com> i always liked the depart mouse. i fit my hand comfortably. and i liked the click and feel of the buttons. it was a good choice by bart locanthi and rob pike to do the mouse that way. andrew hume > > I'll be interested to see what John thinks of the Depraz. > > ron > From ches at cheswick.com Sat Aug 7 05:47:33 2021 From: ches at cheswick.com (William Cheswick) Date: Fri, 6 Aug 2021 15:47:33 -0400 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: <3C6BAD72-7A29-427C-BDE2-AD6636E5EAD7@humeweb.com> References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> <3C6BAD72-7A29-427C-BDE2-AD6636E5EAD7@humeweb.com> Message-ID: <8DFB6C43-BFA5-44F6-928A-495449BB5D2B@cheswick.com> > On Aug 6, 2021, at 3:16 PM, Andrew Hume wrote: > > i always liked the depart mouse. i fit my hand comfortably. > and i liked the click and feel of the buttons. > > it was a good choice by bart locanthi and rob pike to do the mouse that way. > > andrew hume +1. I think there is a market for custom-fitted mice carved out of fancy wood or various metals. The basic shape should follow the Depraz mouse. Maybe a good idea for Etsy. ches From john at jfloren.net Sat Aug 7 07:49:18 2021 From: john at jfloren.net (John Floren) Date: Fri, 06 Aug 2021 21:49:18 +0000 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> Message-ID: I uploaded the high-resolution one to https://jfloren.net/content/unix_skeleton.jpg if anyone wants to check it out in all its glory. Thanks, Rob, this is a great picture. I don't think things were *too* different by the time I visited for IWP9 in 2007, but it's been a long time and I guess I didn't take any pictures then. john ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, August 6th, 2021 at 2:44 PM, Rob Pike wrote: > I sent a higher-res version in which you can read all the text but it was "moderated". > > This is the Unix room as of the year 2000 or so. > > -rob > > On Sat, Aug 7, 2021 at 4:34 AM ron minnich wrote: > >> The story of the mice, one of which I gave to John: >> >> I ran a program called FAST-OS for LANL/Sandia for 6 years starting >> 2005. Think of it as "Plan 9 on petaflop supercomputers" -- it may >> seem strange now, but in that era when some top end systems ran custom >> kernels, there was a strong case to be made that plan 9 was a good >> choice. By 2011, of course, the Linux tsunami had swept all before it, >> which is why you no longer hear about custom HPC kernels so much -- >> though in some places they still reign. In any event, this program >> gave me 6 years to work with "the Unix room", or what was left of it. >> I had been in the Unix Room in 1978, and even met Dennis, so this >> prospect was quite a treat. >> >> We funded Charles Forsyth to write the amd64 compilers for Plan 9, >> which if you used early Go you ran into (6c 6a 6l); we also funded the >> amd64 port of Plan 9 (a.k.a. k10) as well as the port to Blue Gene. >> That amd64 port is still out and about. You can find the Blue Gene >> kernel on github. >> >> I had lots of fun spending time in the Unix room while working with >> the late Jim McKie, and others. I saw the tail end of the traditions. >> They had cookie day once a week, if memory serves, on Thursday at 3. I >> got to see the backwards-running clock, Ken's chess trophies, his >> pilot's license, pictures of Peter everywhere, a "Reagan's view of the >> world" map, the American Legion award for Telstar (which was rescued >> from a dumpster!), and so on. The "Unix room" was more than one room, >> all built on a raised floor, as I assume it was former old school >> machine room space. If memory serves, it filled the entire width of >> the end of the top floor of the building it was in (4th floor?) -- >> maybe 50 ft x 50 ft -- maybe a bit more. There was a room with desks, >> and a similar-sized room with servers, and a smaller room containing a >> lab-style sink, a very professional cappucinno machine, decades of old >> proceedings, and a sofa. I fixed the heavy-duty coffee grinder one >> year; for some reason the Italian company that produced it had seen >> fit to switch BOTH hot and neutral, and the fix was to only switch >> hot, as the neutral switch had failed; I guess in the EU, with 220v, >> things are done differently. >> >> It was fun being there. A few years later the whole room, and all its >> history, was trashed, and replaced with what Jim called a "middle >> management wxx dream" (Jim was never at a loss for words); Jim found >> some yellow Police crime scene tape and placed it in front of the >> doors to the new space. It was redubbed "the innovation space" or some >> such, and looked kind of like an ikea showroom. Much was lost. I tried >> to find a way to save the contents of the room; I had this dream of >> recreating it at Google, much as John Wanamaker's office was preserved >> in Philadelphia for so many decades, but I was too late. I have no >> idea where the contents are now. Maybe next to the Ark. >> >> One day in 2008 or so jmk took me for a tour of the buildings, and we >> at one point ended up high in the top floor of what I think was >> Building One (since torn down?), in what used to be Lab Supply. Nobody >> was there, and not much supply was there either. Finally somebody >> wandered in, and Jim asked where everyone was. "Oh, they closed lab >> supply, maybe 4 years ago?" >> >> Bell Labs had seen hard times since the Lucent split, and it was clear >> it had not quite recovered, and Lab Supply was just one sign of it. I >> think the saddest thing was seeing the visitor center, which I first >> saw in 1976. In 1976, it was the seat of the Bell System Empire, and >> it was huge. There was a map of the US with a light lit for every >> switching office in the Bell Labs system. There was all kinds of Bell >> Labs history in the visitor center museum. >> >> The museum had shrunk to a much smaller area, and felt like a closet. >> The original transistor was still there in 2010, but little else.The >> library was, similarly, changed: it was dark and empty, I was told. >> Money was saved. At that time, Bell Labs felt large, strangely quiet, >> and emptied of people. It made me think of post-sack Rome, ca. 600, >> when its population was estimated to be 500. I have not been back >> since 2011 so maybe things are very different. It would be nice if so. >> >> As part of this tour, Jim gave me 3 depraz mice. I took one, gutted >> it, (sorry!), and filled its guts with a USB mouse innards, and gave >> it back to Jim. He then had a Depraz USB mouse. jmk's mouse did not >> have any lead in it, as John's did, however. The second I gave to >> someone at Google who had worked at the labs back in the day. The >> third mouse I gave to John, and he made it live again, which is cool. >> >> In spite of their reputation, I found Depraz mice hard to use. I have >> gone through all kinds of mice, and am on an evoluent, and as far as >> Depraz go, I guess "you had to be there". I don't recall if jmk used >> his "usb depraz" or it ended up on a shelf. Sadly, I can no longer ask >> him. >> >> I'll be interested to see what John thinks of the Depraz. >> >> ron >> >> On Fri, Aug 6, 2021 at 9:52 AM John Floren wrote: >>> >>> Ah, right. I opened the mouse because one of the encoders didn't seem to be working (it worked fine again this morning, who knows...) and discovered that there was something duct taped inside the plastic shell: >>> >>> http://jfloren.net/content/depraz/inside.jpg >>> >>> Peeling back the tape, I saw what I first took to be chunks of flattened beer cans: >>> >>> http://jfloren.net/content/depraz/reveal.jpg >>> >>> A closer look showed that they were the wrappers which cover the corks of wine bottles. Up into the 1980s, these were made out of lead, and by flattening five of them, a previous owner of the mouse was able to add quite a bit of extra weight to it: >>> >>> http://jfloren.net/content/depraz/wrapper.jpg >>> >>> >>> john >>> >>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >>> >>> On Friday, August 6th, 2021 at 9:34 AM, ron minnich wrote: >>> >>> > john, don't forget to mention the beer can >>> > >>> > On Fri, Aug 6, 2021 at 9:29 AM John Floren john at jfloren.net wrote: >>> > >>> > > I stuck an Arduino on it and with surprisingly little code I have it acting like a 3-button USB mouse. >>> > > >>> > > The only problem is that the pointer doesn't move smoothly. It does OK left-to-right, and can move down pretty well, but going up is a problem. I think pushing the mouse forward tends to move the ball away from the Y-axis wheel, and the old spring on the tensioner just doesn't have the gumption to hold that heavy ball bearing in any more. >>> > > >>> > > john >>> > > >>> > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >>> > > >>> > > On Wednesday, August 4th, 2021 at 9:12 PM, ron minnich rminnich at gmail.com wrote: >>> > > >>> > > > John, you can see that "stick a bird on it" -> "stick an arduino on >>> > > > >>> > > > it" -> "stick a pi on it" has gone as you once predicted :-) >>> > > > >>> > > > On Wed, Aug 4, 2021 at 8:59 PM John Floren john at jfloren.net wrote: >>> > > > >>> > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >>> > > > > >>> > > > > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent henry.r.bent at gmail.com wrote: >>> > > > > >>> > > > > > On Wed, 4 Aug 2021 at 20:52, John Floren john at jfloren.net wrote: >>> > > > > > >>> > > > > > > Having just been given a Depraz mouse, I thought it would be fun to get it working on my modern computer. Since the DE9 connector is male rather than female as you usually see with serial mice, and given its age, I speculate that it might have a custom protocol; in any rate, plugging it into a USB-serial converter and and firing up picocom has given me nothing. >>> > > > > > > >>> > > > > > > Does anyone have a copy of a manual for it, or more information on how to interface with it? If I knew how it was wired and what the protocol looked like, I expect I could make an adapter pretty trivially using a microcontroller. >>> > > > > > >>> > > > > > This might be of some help? >>> > > > > > >>> > > > > > https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 >>> > > > > > >>> > > > > > -Henry >>> > > > > >>> > > > > This looks great, thank you! >>> > > > > >>> > > > > john -------------- next part -------------- An HTML attachment was scrubbed... URL: From robpike at gmail.com Sat Aug 7 07:53:48 2021 From: robpike at gmail.com (Rob Pike) Date: Sat, 7 Aug 2021 07:53:48 +1000 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> Message-ID: I sent a picture (actually two at different resolutions; keep reading) to the list, but being images they are larger than the address space of a PDP-11 so not allowed here. Is it really necessary to have such a low message size limit in an era when I can buy a terabyte of storage for less than a hundred bucks? Here is a Google Drive link, for the adventurous. 20180123-UnixSkeleton.jpg -rob On Sat, Aug 7, 2021 at 7:44 AM Rob Pike wrote: > I sent a higher-res version in which you can read all the text but it was > "moderated". > > This is the Unix room as of the year 2000 or so. > > -rob > > > On Sat, Aug 7, 2021 at 4:34 AM ron minnich wrote: > >> The story of the mice, one of which I gave to John: >> >> I ran a program called FAST-OS for LANL/Sandia for 6 years starting >> 2005. Think of it as "Plan 9 on petaflop supercomputers" -- it may >> seem strange now, but in that era when some top end systems ran custom >> kernels, there was a strong case to be made that plan 9 was a good >> choice. By 2011, of course, the Linux tsunami had swept all before it, >> which is why you no longer hear about custom HPC kernels so much -- >> though in some places they still reign. In any event, this program >> gave me 6 years to work with "the Unix room", or what was left of it. >> I had been in the Unix Room in 1978, and even met Dennis, so this >> prospect was quite a treat. >> >> We funded Charles Forsyth to write the amd64 compilers for Plan 9, >> which if you used early Go you ran into (6c 6a 6l); we also funded the >> amd64 port of Plan 9 (a.k.a. k10) as well as the port to Blue Gene. >> That amd64 port is still out and about. You can find the Blue Gene >> kernel on github. >> >> I had lots of fun spending time in the Unix room while working with >> the late Jim McKie, and others. I saw the tail end of the traditions. >> They had cookie day once a week, if memory serves, on Thursday at 3. I >> got to see the backwards-running clock, Ken's chess trophies, his >> pilot's license, pictures of Peter everywhere, a "Reagan's view of the >> world" map, the American Legion award for Telstar (which was rescued >> from a dumpster!), and so on. The "Unix room" was more than one room, >> all built on a raised floor, as I assume it was former old school >> machine room space. If memory serves, it filled the entire width of >> the end of the top floor of the building it was in (4th floor?) -- >> maybe 50 ft x 50 ft -- maybe a bit more. There was a room with desks, >> and a similar-sized room with servers, and a smaller room containing a >> lab-style sink, a very professional cappucinno machine, decades of old >> proceedings, and a sofa. I fixed the heavy-duty coffee grinder one >> year; for some reason the Italian company that produced it had seen >> fit to switch BOTH hot and neutral, and the fix was to only switch >> hot, as the neutral switch had failed; I guess in the EU, with 220v, >> things are done differently. >> >> It was fun being there. A few years later the whole room, and all its >> history, was trashed, and replaced with what Jim called a "middle >> management wxx dream" (Jim was never at a loss for words); Jim found >> some yellow Police crime scene tape and placed it in front of the >> doors to the new space. It was redubbed "the innovation space" or some >> such, and looked kind of like an ikea showroom. Much was lost. I tried >> to find a way to save the contents of the room; I had this dream of >> recreating it at Google, much as John Wanamaker's office was preserved >> in Philadelphia for so many decades, but I was too late. I have no >> idea where the contents are now. Maybe next to the Ark. >> >> One day in 2008 or so jmk took me for a tour of the buildings, and we >> at one point ended up high in the top floor of what I think was >> Building One (since torn down?), in what used to be Lab Supply. Nobody >> was there, and not much supply was there either. Finally somebody >> wandered in, and Jim asked where everyone was. "Oh, they closed lab >> supply, maybe 4 years ago?" >> >> Bell Labs had seen hard times since the Lucent split, and it was clear >> it had not quite recovered, and Lab Supply was just one sign of it. I >> think the saddest thing was seeing the visitor center, which I first >> saw in 1976. In 1976, it was the seat of the Bell System Empire, and >> it was huge. There was a map of the US with a light lit for every >> switching office in the Bell Labs system. There was all kinds of Bell >> Labs history in the visitor center museum. >> >> The museum had shrunk to a much smaller area, and felt like a closet. >> The original transistor was still there in 2010, but little else.The >> library was, similarly, changed: it was dark and empty, I was told. >> Money was saved. At that time, Bell Labs felt large, strangely quiet, >> and emptied of people. It made me think of post-sack Rome, ca. 600, >> when its population was estimated to be 500. I have not been back >> since 2011 so maybe things are very different. It would be nice if so. >> >> As part of this tour, Jim gave me 3 depraz mice. I took one, gutted >> it, (sorry!), and filled its guts with a USB mouse innards, and gave >> it back to Jim. He then had a Depraz USB mouse. jmk's mouse did not >> have any lead in it, as John's did, however. The second I gave to >> someone at Google who had worked at the labs back in the day. The >> third mouse I gave to John, and he made it live again, which is cool. >> >> In spite of their reputation, I found Depraz mice hard to use. I have >> gone through all kinds of mice, and am on an evoluent, and as far as >> Depraz go, I guess "you had to be there". I don't recall if jmk used >> his "usb depraz" or it ended up on a shelf. Sadly, I can no longer ask >> him. >> >> I'll be interested to see what John thinks of the Depraz. >> >> ron >> >> On Fri, Aug 6, 2021 at 9:52 AM John Floren wrote: >> > >> > Ah, right. I opened the mouse because one of the encoders didn't seem >> to be working (it worked fine again this morning, who knows...) and >> discovered that there was something duct taped inside the plastic shell: >> > >> > http://jfloren.net/content/depraz/inside.jpg >> > >> > Peeling back the tape, I saw what I first took to be chunks of >> flattened beer cans: >> > >> > http://jfloren.net/content/depraz/reveal.jpg >> > >> > A closer look showed that they were the wrappers which cover the corks >> of wine bottles. Up into the 1980s, these were made out of lead, and by >> flattening five of them, a previous owner of the mouse was able to add >> quite a bit of extra weight to it: >> > >> > http://jfloren.net/content/depraz/wrapper.jpg >> > >> > >> > john >> > >> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> > >> > On Friday, August 6th, 2021 at 9:34 AM, ron minnich >> wrote: >> > >> > > john, don't forget to mention the beer can >> > > >> > > On Fri, Aug 6, 2021 at 9:29 AM John Floren john at jfloren.net wrote: >> > > >> > > > I stuck an Arduino on it and with surprisingly little code I have >> it acting like a 3-button USB mouse. >> > > > >> > > > The only problem is that the pointer doesn't move smoothly. It does >> OK left-to-right, and can move down pretty well, but going up is a problem. >> I think pushing the mouse forward tends to move the ball away from the >> Y-axis wheel, and the old spring on the tensioner just doesn't have the >> gumption to hold that heavy ball bearing in any more. >> > > > >> > > > john >> > > > >> > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> > > > >> > > > On Wednesday, August 4th, 2021 at 9:12 PM, ron minnich >> rminnich at gmail.com wrote: >> > > > >> > > > > John, you can see that "stick a bird on it" -> "stick an arduino >> on >> > > > > >> > > > > it" -> "stick a pi on it" has gone as you once predicted :-) >> > > > > >> > > > > On Wed, Aug 4, 2021 at 8:59 PM John Floren john at jfloren.net >> wrote: >> > > > > >> > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> > > > > > >> > > > > > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent >> henry.r.bent at gmail.com wrote: >> > > > > > >> > > > > > > On Wed, 4 Aug 2021 at 20:52, John Floren john at jfloren.net >> wrote: >> > > > > > > >> > > > > > > > Having just been given a Depraz mouse, I thought it would >> be fun to get it working on my modern computer. Since the DE9 connector is >> male rather than female as you usually see with serial mice, and given its >> age, I speculate that it might have a custom protocol; in any rate, >> plugging it into a USB-serial converter and and firing up picocom has given >> me nothing. >> > > > > > > > >> > > > > > > > Does anyone have a copy of a manual for it, or more >> information on how to interface with it? If I knew how it was wired and >> what the protocol looked like, I expect I could make an adapter pretty >> trivially using a microcontroller. >> > > > > > > >> > > > > > > This might be of some help? >> > > > > > > >> > > > > > > >> https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 >> > > > > > > >> > > > > > > -Henry >> > > > > > >> > > > > > This looks great, thank you! >> > > > > > >> > > > > > john >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robpike at gmail.com Sat Aug 7 07:54:35 2021 From: robpike at gmail.com (Rob Pike) Date: Sat, 7 Aug 2021 07:54:35 +1000 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> Message-ID: Thanks - our messages crossed in the post. I keep forgetting that images are verboten here, because it makes no sense to me. -rob On Sat, Aug 7, 2021 at 7:51 AM John Floren wrote: > I uploaded the high-resolution one to > https://jfloren.net/content/unix_skeleton.jpg if anyone wants to check it > out in all its glory. > > Thanks, Rob, this is a great picture. I don't think things were *too* > different by the time I visited for IWP9 in 2007, but it's been a long time > and I guess I didn't take any pictures then. > > john > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > On Friday, August 6th, 2021 at 2:44 PM, Rob Pike > wrote: > > I sent a higher-res version in which you can read all the text but it was > "moderated". > > This is the Unix room as of the year 2000 or so. > > -rob > > > On Sat, Aug 7, 2021 at 4:34 AM ron minnich wrote: > >> The story of the mice, one of which I gave to John: >> >> I ran a program called FAST-OS for LANL/Sandia for 6 years starting >> 2005. Think of it as "Plan 9 on petaflop supercomputers" -- it may >> seem strange now, but in that era when some top end systems ran custom >> kernels, there was a strong case to be made that plan 9 was a good >> choice. By 2011, of course, the Linux tsunami had swept all before it, >> which is why you no longer hear about custom HPC kernels so much -- >> though in some places they still reign. In any event, this program >> gave me 6 years to work with "the Unix room", or what was left of it. >> I had been in the Unix Room in 1978, and even met Dennis, so this >> prospect was quite a treat. >> >> We funded Charles Forsyth to write the amd64 compilers for Plan 9, >> which if you used early Go you ran into (6c 6a 6l); we also funded the >> amd64 port of Plan 9 (a.k.a. k10) as well as the port to Blue Gene. >> That amd64 port is still out and about. You can find the Blue Gene >> kernel on github. >> >> I had lots of fun spending time in the Unix room while working with >> the late Jim McKie, and others. I saw the tail end of the traditions. >> They had cookie day once a week, if memory serves, on Thursday at 3. I >> got to see the backwards-running clock, Ken's chess trophies, his >> pilot's license, pictures of Peter everywhere, a "Reagan's view of the >> world" map, the American Legion award for Telstar (which was rescued >> from a dumpster!), and so on. The "Unix room" was more than one room, >> all built on a raised floor, as I assume it was former old school >> machine room space. If memory serves, it filled the entire width of >> the end of the top floor of the building it was in (4th floor?) -- >> maybe 50 ft x 50 ft -- maybe a bit more. There was a room with desks, >> and a similar-sized room with servers, and a smaller room containing a >> lab-style sink, a very professional cappucinno machine, decades of old >> proceedings, and a sofa. I fixed the heavy-duty coffee grinder one >> year; for some reason the Italian company that produced it had seen >> fit to switch BOTH hot and neutral, and the fix was to only switch >> hot, as the neutral switch had failed; I guess in the EU, with 220v, >> things are done differently. >> >> It was fun being there. A few years later the whole room, and all its >> history, was trashed, and replaced with what Jim called a "middle >> management wxx dream" (Jim was never at a loss for words); Jim found >> some yellow Police crime scene tape and placed it in front of the >> doors to the new space. It was redubbed "the innovation space" or some >> such, and looked kind of like an ikea showroom. Much was lost. I tried >> to find a way to save the contents of the room; I had this dream of >> recreating it at Google, much as John Wanamaker's office was preserved >> in Philadelphia for so many decades, but I was too late. I have no >> idea where the contents are now. Maybe next to the Ark. >> >> One day in 2008 or so jmk took me for a tour of the buildings, and we >> at one point ended up high in the top floor of what I think was >> Building One (since torn down?), in what used to be Lab Supply. Nobody >> was there, and not much supply was there either. Finally somebody >> wandered in, and Jim asked where everyone was. "Oh, they closed lab >> supply, maybe 4 years ago?" >> >> Bell Labs had seen hard times since the Lucent split, and it was clear >> it had not quite recovered, and Lab Supply was just one sign of it. I >> think the saddest thing was seeing the visitor center, which I first >> saw in 1976. In 1976, it was the seat of the Bell System Empire, and >> it was huge. There was a map of the US with a light lit for every >> switching office in the Bell Labs system. There was all kinds of Bell >> Labs history in the visitor center museum. >> >> The museum had shrunk to a much smaller area, and felt like a closet. >> The original transistor was still there in 2010, but little else.The >> library was, similarly, changed: it was dark and empty, I was told. >> Money was saved. At that time, Bell Labs felt large, strangely quiet, >> and emptied of people. It made me think of post-sack Rome, ca. 600, >> when its population was estimated to be 500. I have not been back >> since 2011 so maybe things are very different. It would be nice if so. >> >> As part of this tour, Jim gave me 3 depraz mice. I took one, gutted >> it, (sorry!), and filled its guts with a USB mouse innards, and gave >> it back to Jim. He then had a Depraz USB mouse. jmk's mouse did not >> have any lead in it, as John's did, however. The second I gave to >> someone at Google who had worked at the labs back in the day. The >> third mouse I gave to John, and he made it live again, which is cool. >> >> In spite of their reputation, I found Depraz mice hard to use. I have >> gone through all kinds of mice, and am on an evoluent, and as far as >> Depraz go, I guess "you had to be there". I don't recall if jmk used >> his "usb depraz" or it ended up on a shelf. Sadly, I can no longer ask >> him. >> >> I'll be interested to see what John thinks of the Depraz. >> >> ron >> >> On Fri, Aug 6, 2021 at 9:52 AM John Floren wrote: >> > >> > Ah, right. I opened the mouse because one of the encoders didn't seem >> to be working (it worked fine again this morning, who knows...) and >> discovered that there was something duct taped inside the plastic shell: >> > >> > http://jfloren.net/content/depraz/inside.jpg >> > >> > Peeling back the tape, I saw what I first took to be chunks of >> flattened beer cans: >> > >> > http://jfloren.net/content/depraz/reveal.jpg >> > >> > A closer look showed that they were the wrappers which cover the corks >> of wine bottles. Up into the 1980s, these were made out of lead, and by >> flattening five of them, a previous owner of the mouse was able to add >> quite a bit of extra weight to it: >> > >> > http://jfloren.net/content/depraz/wrapper.jpg >> > >> > >> > john >> > >> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> > >> > On Friday, August 6th, 2021 at 9:34 AM, ron minnich >> wrote: >> > >> > > john, don't forget to mention the beer can >> > > >> > > On Fri, Aug 6, 2021 at 9:29 AM John Floren john at jfloren.net wrote: >> > > >> > > > I stuck an Arduino on it and with surprisingly little code I have >> it acting like a 3-button USB mouse. >> > > > >> > > > The only problem is that the pointer doesn't move smoothly. It does >> OK left-to-right, and can move down pretty well, but going up is a problem. >> I think pushing the mouse forward tends to move the ball away from the >> Y-axis wheel, and the old spring on the tensioner just doesn't have the >> gumption to hold that heavy ball bearing in any more. >> > > > >> > > > john >> > > > >> > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> > > > >> > > > On Wednesday, August 4th, 2021 at 9:12 PM, ron minnich >> rminnich at gmail.com wrote: >> > > > >> > > > > John, you can see that "stick a bird on it" -> "stick an arduino >> on >> > > > > >> > > > > it" -> "stick a pi on it" has gone as you once predicted :-) >> > > > > >> > > > > On Wed, Aug 4, 2021 at 8:59 PM John Floren john at jfloren.net >> wrote: >> > > > > >> > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> > > > > > >> > > > > > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent >> henry.r.bent at gmail.com wrote: >> > > > > > >> > > > > > > On Wed, 4 Aug 2021 at 20:52, John Floren john at jfloren.net >> wrote: >> > > > > > > >> > > > > > > > Having just been given a Depraz mouse, I thought it would >> be fun to get it working on my modern computer. Since the DE9 connector is >> male rather than female as you usually see with serial mice, and given its >> age, I speculate that it might have a custom protocol; in any rate, >> plugging it into a USB-serial converter and and firing up picocom has given >> me nothing. >> > > > > > > > >> > > > > > > > Does anyone have a copy of a manual for it, or more >> information on how to interface with it? If I knew how it was wired and >> what the protocol looked like, I expect I could make an adapter pretty >> trivially using a microcontroller. >> > > > > > > >> > > > > > > This might be of some help? >> > > > > > > >> > > > > > > >> https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 >> > > > > > > >> > > > > > > -Henry >> > > > > > >> > > > > > This looks great, thank you! >> > > > > > >> > > > > > john >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cerise-tuhs at hockeyphil.net Sat Aug 7 09:33:53 2021 From: cerise-tuhs at hockeyphil.net (Phil White) Date: Fri, 6 Aug 2021 16:33:53 -0700 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> Message-ID: <20210806233353.GZ2781665@boogeyman.armory.com> I'm a little embarrassed to ask, but my curiosity demands I ask. Who is that in the framed photograph near the ceiling and between the "Protect Your Password" and "UNIX International Member" posters? -Phil On Sat, Aug 07, 2021 at 07:53:48AM +1000, Rob Pike wrote: > I sent a picture (actually two at different resolutions; keep reading) to > the list, but being images they are larger than the address space of a > PDP-11 so not allowed here. > > Is it really necessary to have such a low message size limit in an era when > I can buy a terabyte of storage for less than a hundred bucks? > > Here is a Google Drive link, for the adventurous. > > > 20180123-UnixSkeleton.jpg > > > > -rob > > > On Sat, Aug 7, 2021 at 7:44 AM Rob Pike wrote: > > > I sent a higher-res version in which you can read all the text but it was > > "moderated". > > > > This is the Unix room as of the year 2000 or so. > > > > -rob > > > > > > On Sat, Aug 7, 2021 at 4:34 AM ron minnich wrote: > > > >> The story of the mice, one of which I gave to John: > >> > >> I ran a program called FAST-OS for LANL/Sandia for 6 years starting > >> 2005. Think of it as "Plan 9 on petaflop supercomputers" -- it may > >> seem strange now, but in that era when some top end systems ran custom > >> kernels, there was a strong case to be made that plan 9 was a good > >> choice. By 2011, of course, the Linux tsunami had swept all before it, > >> which is why you no longer hear about custom HPC kernels so much -- > >> though in some places they still reign. In any event, this program > >> gave me 6 years to work with "the Unix room", or what was left of it. > >> I had been in the Unix Room in 1978, and even met Dennis, so this > >> prospect was quite a treat. > >> > >> We funded Charles Forsyth to write the amd64 compilers for Plan 9, > >> which if you used early Go you ran into (6c 6a 6l); we also funded the > >> amd64 port of Plan 9 (a.k.a. k10) as well as the port to Blue Gene. > >> That amd64 port is still out and about. You can find the Blue Gene > >> kernel on github. > >> > >> I had lots of fun spending time in the Unix room while working with > >> the late Jim McKie, and others. I saw the tail end of the traditions. > >> They had cookie day once a week, if memory serves, on Thursday at 3. I > >> got to see the backwards-running clock, Ken's chess trophies, his > >> pilot's license, pictures of Peter everywhere, a "Reagan's view of the > >> world" map, the American Legion award for Telstar (which was rescued > >> from a dumpster!), and so on. The "Unix room" was more than one room, > >> all built on a raised floor, as I assume it was former old school > >> machine room space. If memory serves, it filled the entire width of > >> the end of the top floor of the building it was in (4th floor?) -- > >> maybe 50 ft x 50 ft -- maybe a bit more. There was a room with desks, > >> and a similar-sized room with servers, and a smaller room containing a > >> lab-style sink, a very professional cappucinno machine, decades of old > >> proceedings, and a sofa. I fixed the heavy-duty coffee grinder one > >> year; for some reason the Italian company that produced it had seen > >> fit to switch BOTH hot and neutral, and the fix was to only switch > >> hot, as the neutral switch had failed; I guess in the EU, with 220v, > >> things are done differently. > >> > >> It was fun being there. A few years later the whole room, and all its > >> history, was trashed, and replaced with what Jim called a "middle > >> management wxx dream" (Jim was never at a loss for words); Jim found > >> some yellow Police crime scene tape and placed it in front of the > >> doors to the new space. It was redubbed "the innovation space" or some > >> such, and looked kind of like an ikea showroom. Much was lost. I tried > >> to find a way to save the contents of the room; I had this dream of > >> recreating it at Google, much as John Wanamaker's office was preserved > >> in Philadelphia for so many decades, but I was too late. I have no > >> idea where the contents are now. Maybe next to the Ark. > >> > >> One day in 2008 or so jmk took me for a tour of the buildings, and we > >> at one point ended up high in the top floor of what I think was > >> Building One (since torn down?), in what used to be Lab Supply. Nobody > >> was there, and not much supply was there either. Finally somebody > >> wandered in, and Jim asked where everyone was. "Oh, they closed lab > >> supply, maybe 4 years ago?" > >> > >> Bell Labs had seen hard times since the Lucent split, and it was clear > >> it had not quite recovered, and Lab Supply was just one sign of it. I > >> think the saddest thing was seeing the visitor center, which I first > >> saw in 1976. In 1976, it was the seat of the Bell System Empire, and > >> it was huge. There was a map of the US with a light lit for every > >> switching office in the Bell Labs system. There was all kinds of Bell > >> Labs history in the visitor center museum. > >> > >> The museum had shrunk to a much smaller area, and felt like a closet. > >> The original transistor was still there in 2010, but little else.The > >> library was, similarly, changed: it was dark and empty, I was told. > >> Money was saved. At that time, Bell Labs felt large, strangely quiet, > >> and emptied of people. It made me think of post-sack Rome, ca. 600, > >> when its population was estimated to be 500. I have not been back > >> since 2011 so maybe things are very different. It would be nice if so. > >> > >> As part of this tour, Jim gave me 3 depraz mice. I took one, gutted > >> it, (sorry!), and filled its guts with a USB mouse innards, and gave > >> it back to Jim. He then had a Depraz USB mouse. jmk's mouse did not > >> have any lead in it, as John's did, however. The second I gave to > >> someone at Google who had worked at the labs back in the day. The > >> third mouse I gave to John, and he made it live again, which is cool. > >> > >> In spite of their reputation, I found Depraz mice hard to use. I have > >> gone through all kinds of mice, and am on an evoluent, and as far as > >> Depraz go, I guess "you had to be there". I don't recall if jmk used > >> his "usb depraz" or it ended up on a shelf. Sadly, I can no longer ask > >> him. > >> > >> I'll be interested to see what John thinks of the Depraz. > >> > >> ron > >> > >> On Fri, Aug 6, 2021 at 9:52 AM John Floren wrote: > >> > > >> > Ah, right. I opened the mouse because one of the encoders didn't seem > >> to be working (it worked fine again this morning, who knows...) and > >> discovered that there was something duct taped inside the plastic shell: > >> > > >> > http://jfloren.net/content/depraz/inside.jpg > >> > > >> > Peeling back the tape, I saw what I first took to be chunks of > >> flattened beer cans: > >> > > >> > http://jfloren.net/content/depraz/reveal.jpg > >> > > >> > A closer look showed that they were the wrappers which cover the corks > >> of wine bottles. Up into the 1980s, these were made out of lead, and by > >> flattening five of them, a previous owner of the mouse was able to add > >> quite a bit of extra weight to it: > >> > > >> > http://jfloren.net/content/depraz/wrapper.jpg > >> > > >> > > >> > john > >> > > >> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > >> > > >> > On Friday, August 6th, 2021 at 9:34 AM, ron minnich > >> wrote: > >> > > >> > > john, don't forget to mention the beer can > >> > > > >> > > On Fri, Aug 6, 2021 at 9:29 AM John Floren john at jfloren.net wrote: > >> > > > >> > > > I stuck an Arduino on it and with surprisingly little code I have > >> it acting like a 3-button USB mouse. > >> > > > > >> > > > The only problem is that the pointer doesn't move smoothly. It does > >> OK left-to-right, and can move down pretty well, but going up is a problem. > >> I think pushing the mouse forward tends to move the ball away from the > >> Y-axis wheel, and the old spring on the tensioner just doesn't have the > >> gumption to hold that heavy ball bearing in any more. > >> > > > > >> > > > john > >> > > > > >> > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > >> > > > > >> > > > On Wednesday, August 4th, 2021 at 9:12 PM, ron minnich > >> rminnich at gmail.com wrote: > >> > > > > >> > > > > John, you can see that "stick a bird on it" -> "stick an arduino > >> on > >> > > > > > >> > > > > it" -> "stick a pi on it" has gone as you once predicted :-) > >> > > > > > >> > > > > On Wed, Aug 4, 2021 at 8:59 PM John Floren john at jfloren.net > >> wrote: > >> > > > > > >> > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > >> > > > > > > >> > > > > > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent > >> henry.r.bent at gmail.com wrote: > >> > > > > > > >> > > > > > > On Wed, 4 Aug 2021 at 20:52, John Floren john at jfloren.net > >> wrote: > >> > > > > > > > >> > > > > > > > Having just been given a Depraz mouse, I thought it would > >> be fun to get it working on my modern computer. Since the DE9 connector is > >> male rather than female as you usually see with serial mice, and given its > >> age, I speculate that it might have a custom protocol; in any rate, > >> plugging it into a USB-serial converter and and firing up picocom has given > >> me nothing. > >> > > > > > > > > >> > > > > > > > Does anyone have a copy of a manual for it, or more > >> information on how to interface with it? If I knew how it was wired and > >> what the protocol looked like, I expect I could make an adapter pretty > >> trivially using a microcontroller. > >> > > > > > > > >> > > > > > > This might be of some help? > >> > > > > > > > >> > > > > > > > >> https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 > >> > > > > > > > >> > > > > > > -Henry > >> > > > > > > >> > > > > > This looks great, thank you! > >> > > > > > > >> > > > > > john > >> > > From norman at oclsc.org Sat Aug 7 09:48:36 2021 From: norman at oclsc.org (Norman Wilson) Date: Fri, 6 Aug 2021 19:48:36 -0400 (EDT) Subject: [TUHS] Depraz/Logitech Digimouse manual Message-ID: <20210806234836.B720C640CC6@lignose.oclsc.org> The mouse with wine-bottle lead foil in the top may have been my fault. I did that to two of them--at home and in my office--because I found a little more pressure made the ball track better. I've never been an alcohol-consumer; the lead came from a friend, who used to save it (back in the 1980s) to mail to Republicans. Apparently he had, many years before, registered to vote in a Republican primary solely to oppose a particularly-poor candidate. That somehow got him on a GOP mailing list that sent him endless funding appeals with post-paid envelopes. He used to fill the envelopes with lead and drop them in the mail, in the hope that he would cost the party even more in excess postage than they were already spending to send the funding pitches. By the time I was at Bell Labs, he had moved to Canada, and was no longer receiving unwanted political funding pitches, but he was glad to save a few bits of lead for me when I thought of the trick and asked him. Only too glad, it turned out; he kept saving it and saving it and saving it even though I neither needed nor wanted any more. Eventually I managed to get the message through to him. He has since moved back to the US. He is still fond of wine. I don't know what he does with the cork wrappers. Norman Wilson Toronto ON From cowan at ccil.org Sat Aug 7 09:57:21 2021 From: cowan at ccil.org (John Cowan) Date: Fri, 6 Aug 2021 19:57:21 -0400 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: <20210806234836.B720C640CC6@lignose.oclsc.org> References: <20210806234836.B720C640CC6@lignose.oclsc.org> Message-ID: On Fri, Aug 6, 2021 at 7:49 PM Norman Wilson wrote: He used to > fill the envelopes with lead and drop them in the mail, > in the hope that he would cost the party even more in > excess postage than they were already spending to send > the funding pitches. > Piker. *Real* practitioners of this sport would attach a business-reply label (basically a postcard you can affix to a package) to a *brick*, thus easily sticking the recipient for US$5 or more, of course worth a lot more backintheday. For many years now, though, anything like either case is officially classified as "waste", which means the USPS throws it out before it ever gets anywhere, same as happens when you dump your trash into a mailbox. I suppose Canada Post does the same. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robpike at gmail.com Sat Aug 7 10:24:14 2021 From: robpike at gmail.com (Rob Pike) Date: Sat, 7 Aug 2021 10:24:14 +1000 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: <20210806233353.GZ2781665@boogeyman.armory.com> References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> <20210806233353.GZ2781665@boogeyman.armory.com> Message-ID: I don't think anyone knows. Nobody relevant, I believe. -rob On Sat, Aug 7, 2021 at 9:33 AM Phil White wrote: > I'm a little embarrassed to ask, but my curiosity demands I ask. Who is > that in the framed photograph near the ceiling and between the "Protect > Your Password" and "UNIX International Member" posters? > > -Phil > > On Sat, Aug 07, 2021 at 07:53:48AM +1000, Rob Pike wrote: > > I sent a picture (actually two at different resolutions; keep reading) to > > the list, but being images they are larger than the address space of a > > PDP-11 so not allowed here. > > > > Is it really necessary to have such a low message size limit in an era > when > > I can buy a terabyte of storage for less than a hundred bucks? > > > > Here is a Google Drive link, for the adventurous. > > > > > > 20180123-UnixSkeleton.jpg > > < > https://drive.google.com/file/d/1aS8ZmzwPUawIa8WXGoXOK9jDiYtJETGG/view?usp=drive_web > > > > > > > > -rob > > > > > > On Sat, Aug 7, 2021 at 7:44 AM Rob Pike wrote: > > > > > I sent a higher-res version in which you can read all the text but it > was > > > "moderated". > > > > > > This is the Unix room as of the year 2000 or so. > > > > > > -rob > > > > > > > > > On Sat, Aug 7, 2021 at 4:34 AM ron minnich wrote: > > > > > >> The story of the mice, one of which I gave to John: > > >> > > >> I ran a program called FAST-OS for LANL/Sandia for 6 years starting > > >> 2005. Think of it as "Plan 9 on petaflop supercomputers" -- it may > > >> seem strange now, but in that era when some top end systems ran custom > > >> kernels, there was a strong case to be made that plan 9 was a good > > >> choice. By 2011, of course, the Linux tsunami had swept all before it, > > >> which is why you no longer hear about custom HPC kernels so much -- > > >> though in some places they still reign. In any event, this program > > >> gave me 6 years to work with "the Unix room", or what was left of it. > > >> I had been in the Unix Room in 1978, and even met Dennis, so this > > >> prospect was quite a treat. > > >> > > >> We funded Charles Forsyth to write the amd64 compilers for Plan 9, > > >> which if you used early Go you ran into (6c 6a 6l); we also funded the > > >> amd64 port of Plan 9 (a.k.a. k10) as well as the port to Blue Gene. > > >> That amd64 port is still out and about. You can find the Blue Gene > > >> kernel on github. > > >> > > >> I had lots of fun spending time in the Unix room while working with > > >> the late Jim McKie, and others. I saw the tail end of the traditions. > > >> They had cookie day once a week, if memory serves, on Thursday at 3. I > > >> got to see the backwards-running clock, Ken's chess trophies, his > > >> pilot's license, pictures of Peter everywhere, a "Reagan's view of the > > >> world" map, the American Legion award for Telstar (which was rescued > > >> from a dumpster!), and so on. The "Unix room" was more than one room, > > >> all built on a raised floor, as I assume it was former old school > > >> machine room space. If memory serves, it filled the entire width of > > >> the end of the top floor of the building it was in (4th floor?) -- > > >> maybe 50 ft x 50 ft -- maybe a bit more. There was a room with desks, > > >> and a similar-sized room with servers, and a smaller room containing a > > >> lab-style sink, a very professional cappucinno machine, decades of old > > >> proceedings, and a sofa. I fixed the heavy-duty coffee grinder one > > >> year; for some reason the Italian company that produced it had seen > > >> fit to switch BOTH hot and neutral, and the fix was to only switch > > >> hot, as the neutral switch had failed; I guess in the EU, with 220v, > > >> things are done differently. > > >> > > >> It was fun being there. A few years later the whole room, and all its > > >> history, was trashed, and replaced with what Jim called a "middle > > >> management wxx dream" (Jim was never at a loss for words); Jim found > > >> some yellow Police crime scene tape and placed it in front of the > > >> doors to the new space. It was redubbed "the innovation space" or some > > >> such, and looked kind of like an ikea showroom. Much was lost. I tried > > >> to find a way to save the contents of the room; I had this dream of > > >> recreating it at Google, much as John Wanamaker's office was preserved > > >> in Philadelphia for so many decades, but I was too late. I have no > > >> idea where the contents are now. Maybe next to the Ark. > > >> > > >> One day in 2008 or so jmk took me for a tour of the buildings, and we > > >> at one point ended up high in the top floor of what I think was > > >> Building One (since torn down?), in what used to be Lab Supply. Nobody > > >> was there, and not much supply was there either. Finally somebody > > >> wandered in, and Jim asked where everyone was. "Oh, they closed lab > > >> supply, maybe 4 years ago?" > > >> > > >> Bell Labs had seen hard times since the Lucent split, and it was clear > > >> it had not quite recovered, and Lab Supply was just one sign of it. I > > >> think the saddest thing was seeing the visitor center, which I first > > >> saw in 1976. In 1976, it was the seat of the Bell System Empire, and > > >> it was huge. There was a map of the US with a light lit for every > > >> switching office in the Bell Labs system. There was all kinds of Bell > > >> Labs history in the visitor center museum. > > >> > > >> The museum had shrunk to a much smaller area, and felt like a closet. > > >> The original transistor was still there in 2010, but little else.The > > >> library was, similarly, changed: it was dark and empty, I was told. > > >> Money was saved. At that time, Bell Labs felt large, strangely quiet, > > >> and emptied of people. It made me think of post-sack Rome, ca. 600, > > >> when its population was estimated to be 500. I have not been back > > >> since 2011 so maybe things are very different. It would be nice if so. > > >> > > >> As part of this tour, Jim gave me 3 depraz mice. I took one, gutted > > >> it, (sorry!), and filled its guts with a USB mouse innards, and gave > > >> it back to Jim. He then had a Depraz USB mouse. jmk's mouse did not > > >> have any lead in it, as John's did, however. The second I gave to > > >> someone at Google who had worked at the labs back in the day. The > > >> third mouse I gave to John, and he made it live again, which is cool. > > >> > > >> In spite of their reputation, I found Depraz mice hard to use. I have > > >> gone through all kinds of mice, and am on an evoluent, and as far as > > >> Depraz go, I guess "you had to be there". I don't recall if jmk used > > >> his "usb depraz" or it ended up on a shelf. Sadly, I can no longer ask > > >> him. > > >> > > >> I'll be interested to see what John thinks of the Depraz. > > >> > > >> ron > > >> > > >> On Fri, Aug 6, 2021 at 9:52 AM John Floren wrote: > > >> > > > >> > Ah, right. I opened the mouse because one of the encoders didn't > seem > > >> to be working (it worked fine again this morning, who knows...) and > > >> discovered that there was something duct taped inside the plastic > shell: > > >> > > > >> > http://jfloren.net/content/depraz/inside.jpg > > >> > > > >> > Peeling back the tape, I saw what I first took to be chunks of > > >> flattened beer cans: > > >> > > > >> > http://jfloren.net/content/depraz/reveal.jpg > > >> > > > >> > A closer look showed that they were the wrappers which cover the > corks > > >> of wine bottles. Up into the 1980s, these were made out of lead, and > by > > >> flattening five of them, a previous owner of the mouse was able to add > > >> quite a bit of extra weight to it: > > >> > > > >> > http://jfloren.net/content/depraz/wrapper.jpg > > >> > > > >> > > > >> > john > > >> > > > >> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > >> > > > >> > On Friday, August 6th, 2021 at 9:34 AM, ron minnich < > rminnich at gmail.com> > > >> wrote: > > >> > > > >> > > john, don't forget to mention the beer can > > >> > > > > >> > > On Fri, Aug 6, 2021 at 9:29 AM John Floren john at jfloren.net > wrote: > > >> > > > > >> > > > I stuck an Arduino on it and with surprisingly little code I > have > > >> it acting like a 3-button USB mouse. > > >> > > > > > >> > > > The only problem is that the pointer doesn't move smoothly. It > does > > >> OK left-to-right, and can move down pretty well, but going up is a > problem. > > >> I think pushing the mouse forward tends to move the ball away from the > > >> Y-axis wheel, and the old spring on the tensioner just doesn't have > the > > >> gumption to hold that heavy ball bearing in any more. > > >> > > > > > >> > > > john > > >> > > > > > >> > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > >> > > > > > >> > > > On Wednesday, August 4th, 2021 at 9:12 PM, ron minnich > > >> rminnich at gmail.com wrote: > > >> > > > > > >> > > > > John, you can see that "stick a bird on it" -> "stick an > arduino > > >> on > > >> > > > > > > >> > > > > it" -> "stick a pi on it" has gone as you once predicted :-) > > >> > > > > > > >> > > > > On Wed, Aug 4, 2021 at 8:59 PM John Floren john at jfloren.net > > >> wrote: > > >> > > > > > > >> > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > >> > > > > > > > >> > > > > > On Wednesday, August 4th, 2021 at 6:12 PM, Henry Bent > > >> henry.r.bent at gmail.com wrote: > > >> > > > > > > > >> > > > > > > On Wed, 4 Aug 2021 at 20:52, John Floren john at jfloren.net > > >> wrote: > > >> > > > > > > > > >> > > > > > > > Having just been given a Depraz mouse, I thought it > would > > >> be fun to get it working on my modern computer. Since the DE9 > connector is > > >> male rather than female as you usually see with serial mice, and > given its > > >> age, I speculate that it might have a custom protocol; in any rate, > > >> plugging it into a USB-serial converter and and firing up picocom has > given > > >> me nothing. > > >> > > > > > > > > > >> > > > > > > > Does anyone have a copy of a manual for it, or more > > >> information on how to interface with it? If I knew how it was wired > and > > >> what the protocol looked like, I expect I could make an adapter pretty > > >> trivially using a microcontroller. > > >> > > > > > > > > >> > > > > > > This might be of some help? > > >> > > > > > > > > >> > > > > > > > > >> > https://www.vcfed.org/forum/forum/technical-support/vintage-computer-hardware/74403-whitechapel-mg-1-depraz-mouse-grey-pinout#post904391 > > >> > > > > > > > > >> > > > > > > -Henry > > >> > > > > > > > >> > > > > > This looks great, thank you! > > >> > > > > > > > >> > > > > > john > > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfoust at threedee.com Sat Aug 7 22:13:29 2021 From: jfoust at threedee.com (John Foust) Date: Sat, 07 Aug 2021 07:13:29 -0500 Subject: [TUHS] Depraz/Logitech Digimouse manual In-Reply-To: References: <-AmqyEPDFGA7JKlNEOgAqahmsc7dqEexT1kaQbALEzVtMU9DkZiBCg9p4ikYcJwHGsYprzFx13yPLlbW3PL0lHGBWFcDFp2dcleM7B20wcc=@jfloren.net> <20210806233353.GZ2781665@boogeyman.armory.com> Message-ID: <20210807132018.DB70F9C9E8@minnie.tuhs.org> At 07:24 PM 8/6/2021, Rob Pike wrote: >I don't think anyone knows. Nobody relevant, I believe. Indeed, a clipped and cleaned version in reverse image search on Google, Bing, Tineye and Yandex found nothing. Feels like a head shot for a theater major. - John From ches at cheswick.com Sun Aug 8 02:17:35 2021 From: ches at cheswick.com (William Cheswick) Date: Sat, 7 Aug 2021 12:17:35 -0400 Subject: [TUHS] Who is that in the framed photograph near the ceiling? Message-ID: <37F19C12-1615-4C30-9336-32ABE1740941@cheswick.com> > I don't think anyone knows. Nobody relevant, I believe. > > -rob I understand that Dave Presotto bought that photo at a garage sale for $1. The photo hung in the Unix Room for years, at one point labeled “Peter Weinberger.” One day I removed it from its careful mounting and scanned in the photo. It bore the label “what, no steak?” The photo was stolen from a wall sometime after I left. The scanned image is at https://cheswick.com/ches/tmp/whatnosteak.jpeg ches From steve at quintile.net Mon Aug 9 07:23:39 2021 From: steve at quintile.net (Steve Simon) Date: Sun, 8 Aug 2021 22:23:39 +0100 Subject: [TUHS] Who is that in the framed photograph near the ceiling? Message-ID: <397BA7B9-8AF7-469A-BFB6-B656B9AE4692@quintile.net> great picture. once again i am taken aback at the good taste of the residents of the unix room. -Steve From kennethgoodwin56 at gmail.com Mon Aug 9 07:40:26 2021 From: kennethgoodwin56 at gmail.com (Kenneth Goodwin) Date: Sun, 8 Aug 2021 17:40:26 -0400 Subject: [TUHS] Who is that in the framed photograph near the ceiling? In-Reply-To: <397BA7B9-8AF7-469A-BFB6-B656B9AE4692@quintile.net> References: <397BA7B9-8AF7-469A-BFB6-B656B9AE4692@quintile.net> Message-ID: And their elite dumpster diving skills..... On Sun, Aug 8, 2021, 5:32 PM Steve Simon wrote: > > great picture. > > once again i am taken aback at the good taste of the residents of the unix > room. > > -Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From norman at oclsc.org Mon Aug 9 08:14:56 2021 From: norman at oclsc.org (Norman Wilson) Date: Sun, 8 Aug 2021 18:14:56 -0400 (EDT) Subject: [TUHS] Who is that in the framed photograph near the ceiling? Message-ID: <20210808221456.30FE9640CC6@lignose.oclsc.org> Steve Simon: once again i am taken aback at the good taste of the residents of the unix room. As a whilom denizen of that esteemed playroom, I question both the accuracy and the relevance of that metric. Besides, what happened to the sheep shelf? Was it scrubbed away after I left? And, Ken, whatever happened to Dolly the Sheep (after she was hidden to avoid upsetting visitors)? Norman Wilson Toronto ON No longer a subscriber to sheep! magazine From kenbob at gmail.com Mon Aug 9 10:38:00 2021 From: kenbob at gmail.com (Ken Thompson) Date: Sun, 8 Aug 2021 17:38:00 -0700 Subject: [TUHS] Who is that in the framed photograph near the ceiling? In-Reply-To: <20210808221456.30FE9640CC6@lignose.oclsc.org> References: <20210808221456.30FE9640CC6@lignose.oclsc.org> Message-ID: dolly is at my home in san jose. she says "hi." On Sun, Aug 8, 2021 at 3:17 PM Norman Wilson wrote: > Steve Simon: > > once again i am taken aback at the good taste of the residents of the > unix room. > > As a whilom denizen of that esteemed playroom, I question > both the accuracy and the relevance of that metric. > > Besides, what happened to the sheep shelf? Was it scrubbed > away after I left? And, Ken, whatever happened to Dolly the > Sheep (after she was hidden to avoid upsetting visitors)? > > Norman Wilson > Toronto ON > No longer a subscriber to sheep! magazine > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tih at hamartun.priv.no Thu Aug 12 04:11:28 2021 From: tih at hamartun.priv.no (Tom Ivar Helbekkmo) Date: Wed, 11 Aug 2021 20:11:28 +0200 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: (Tom Lyon via TUHS's message of "Tue, 3 Aug 2021 10:13:31 -0700") References: Message-ID: Tom Lyon via TUHS writes: > What if CPU designers would add facilities to directly implement > inter-process or inter-processor messaging? You mean like INMOS's Transputer architecture from back in the late eighties and early nineties? Each processor had a thread scheduling and message passing microkernel implemented in its microcode, and had four bi-directional links to other processors, so you could build a grid. They designed the language Occam along with it, to be its lowest level language; it and the instruction set were designed to match. Occam has threads and message passing as built-in concepts, of course. -tih (playing with the Helios distributed OS on Transputer hardware) -- Most people who graduate with CS degrees don't understand the significance of Lisp. Lisp is the most important idea in computer science. --Alan Kay From tih at hamartun.priv.no Thu Aug 12 04:11:28 2021 From: tih at hamartun.priv.no (Tom Ivar Helbekkmo) Date: Wed, 11 Aug 2021 20:11:28 +0200 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: (Tom Lyon via TUHS's message of "Tue, 3 Aug 2021 10:13:31 -0700") References: Message-ID: Tom Lyon via TUHS writes: > What if CPU designers would add facilities to directly implement > inter-process or inter-processor messaging? You mean like INMOS's Transputer architecture from back in the late eighties and early nineties? Each processor had a thread scheduling and message passing microkernel implemented in its microcode, and had four bi-directional links to other processors, so you could build a grid. They designed the language Occam along with it, to be its lowest level language; it and the instruction set were designed to match. Occam has threads and message passing as built-in concepts, of course. -tih (playing with the Helios distributed OS on Transputer hardware) -- Most people who graduate with CS degrees don't understand the significance of Lisp. Lisp is the most important idea in computer science. --Alan Kay From pugs at ieee.org Thu Aug 12 07:24:02 2021 From: pugs at ieee.org (Tom Lyon) Date: Wed, 11 Aug 2021 14:24:02 -0700 Subject: [TUHS] Systematic approach to command-line interfaces In-Reply-To: References: Message-ID: I never had the pleasure of playing with Transputers, but it sounds nice. Of course, I'd want the networking to be Ethernet/IP centric these days. On Wed, Aug 11, 2021 at 11:11 AM Tom Ivar Helbekkmo wrote: > Tom Lyon via TUHS writes: > > > What if CPU designers would add facilities to directly implement > > inter-process or inter-processor messaging? > > You mean like INMOS's Transputer architecture from back in the late > eighties and early nineties? Each processor had a thread scheduling and > message passing microkernel implemented in its microcode, and had four > bi-directional links to other processors, so you could build a grid. > They designed the language Occam along with it, to be its lowest level > language; it and the instruction set were designed to match. Occam has > threads and message passing as built-in concepts, of course. > > -tih (playing with the Helios distributed OS on Transputer hardware) > -- > Most people who graduate with CS degrees don't understand the significance > of Lisp. Lisp is the most important idea in computer science. --Alan Kay > -- - Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From douglas.mcilroy at dartmouth.edu Mon Aug 16 11:33:10 2021 From: douglas.mcilroy at dartmouth.edu (Douglas McIlroy) Date: Sun, 15 Aug 2021 21:33:10 -0400 Subject: [TUHS] Can't break line warning Message-ID: This may be due to logic similar to that of a classic feature that I always deemed a bug: troff begins a new page when the current page is exactly filled, rather than waiting until forced by content that doesn't fit. If this condition happens at the end of a document, a spurious blank page results. Worse, if the page header happens to change just after the exactly filled page, the old heading will be produced before the new heading is read. Doug From jon at fourwinds.com Mon Aug 30 08:12:16 2021 From: jon at fourwinds.com (Jon Steinhart) Date: Sun, 29 Aug 2021 15:12:16 -0700 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? Message-ID: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> I recently upgraded my machines to fc34. I just did a stock uncomplicated installation using the defaults and it failed miserably. Fc34 uses btrfs as the default filesystem so I thought that I'd give it a try. I was especially interested in the automatic checksumming because the majority of my storage is large media files and I worry about bit rot in seldom used files. I have been keeping a separate database of file hashes and in theory btrfs would make that automatic and transparent. I have 32T of disk on my system, so it took a long time to convert everything over. A few weeks after I did this I went to unload my camera and couldn't because the filesystem that holds my photos was mounted read-only. WTF? I didn't do that. After a bit of poking around I discovered that btrfs SILENTLY remounted the filesystem because it had errors. Sure, it put something in a log file, but I don't spend all day surfing logs for things that shouldn't be going wrong. Maybe my expectation that filesystems just work is antiquated. This was on a brand new 16T drive, so I didn't think that it was worth the month that it would take to run the badblocks program which doesn't really scale to modern disk sizes. Besides, SMART said that it was fine. Although it's been discredited by some, I'm still a believer in "stop and fsck" policing of disk drives. Unmounted the filesystem and ran fsck to discover that btrfs had to do its own thing. No idea why; I guess some think that incompatibility is a good thing. Ran "btrfs check" which reported errors in the filesystem but was otherwise useless BECAUSE IT DIDN'T FIX ANYTHING. What good is knowing that the filesystem has errors if you can't fix them? Near the top of the manual page it says: Warning Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck successfully repair all types of filesystem corruption. Eg. some other software or hardware bugs can fatally damage a volume. Whoa! I'm sure that operators are standing by, call 1-800-FIX-BTRFS. Really? Is a ploy by the developers to form a support business? Later on, the manual page says: DANGEROUS OPTIONS --repair enable the repair mode and attempt to fix problems where possible Note there’s a warning and 10 second delay when this option is run without --force to give users a chance to think twice before running repair, the warnings in documentation have shown to be insufficient Since when is it dangerous to repair a filesystem? That's a new one to me. Having no option other than not being able to use the disk, I ran btrfs check with the --repair option. It crashed. Lesson so far is that trusting my data to an unreliable unrepairable filesystem is not a good idea. Since this was one of my media disks I just rebuilt it using ext4. Last week I was working away and tried to write out a file to discover that /home and /root had become read-only. Charming. Tried rebooting, but couldn't since btrfs filesystems aren't checked and repaired. Plugged in a flash drive with a live version, managed to successfully run --repair, and rebooted. Lasted about 15 minutes before flipping back to read only with the same error. Time to suck it up and revert. Started a clean reinstall. Got stuck because it crashed during disk setup with anaconda giving me a completely useless big python stack trace. Eventually figured out that it was unable to delete the btrfs filesystem that had errors so it just crashed instead. Wiped it using dd; nice that some reliable tools still survive. Finished the installation and am back up and running. Any of the rest of you have any experiences with btrfs? I'm sure that it works fine at large companies that can afford a team of disk babysitters. What benefits does btrfs provide that other filesystem formats such as ext4 and ZFS don't? Is it just a continuation of the "we have to do everything ourselves and under no circumstances use anything that came from the BSD world" mentality? So what's the future for filesystem repair? Does it look like the past? Is Ken's original need for dsw going to rise from the dead? In my limited experience btrfs is a BiTteR FileSystem to swallow. Or, as Saturday Night Live might put it: And now, linux, starring the not ready for prime time filesystem. Seems like something that's been under development for around 15 years should be in better shape. Jon From henry.r.bent at gmail.com Mon Aug 30 09:09:50 2021 From: henry.r.bent at gmail.com (Henry Bent) Date: Sun, 29 Aug 2021 19:09:50 -0400 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: On Sun, 29 Aug 2021 at 18:13, Jon Steinhart wrote: > I recently upgraded my machines to fc34. I just did a stock > uncomplicated installation using the defaults and it failed miserably. > > Fc34 uses btrfs as the default filesystem so I thought that I'd give it > a try. > ... cut out a lot about how no sane person would want to use btrfs ... > > Or, as Saturday Night Live might put it: And now, linux, starring the > not ready for prime time filesystem. Seems like something that's been > under development for around 15 years should be in better shape. > To my way of thinking this isn't a Linux problem, or even a btrfs problem, it's a Fedora problem. They're the ones who decided to switch their default filesystem to something that clearly isn't ready for prime time. -Henry -------------- next part -------------- An HTML attachment was scrubbed... URL: From lm at mcvoy.com Mon Aug 30 09:57:45 2021 From: lm at mcvoy.com (Larry McVoy) Date: Sun, 29 Aug 2021 16:57:45 -0700 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: <20210829235745.GC20021@mcvoy.com> On Sun, Aug 29, 2021 at 03:12:16PM -0700, Jon Steinhart wrote: > After a bit of poking around I discovered that btrfs SILENTLY remounted the > filesystem because it had errors. Sure, it put something in a log file, > but I don't spend all day surfing logs for things that shouldn't be going > wrong. Maybe my expectation that filesystems just work is antiquated. I give them credit for remounting read-only when seeing errors, they may have gotten that from BitKeeper. When we opened a history file, if we encountered any errors we opened the history file in read only mode so if it worked enough you could see your data, great, but don't write on top of bad data. > Although it's been discredited by some, I'm still a believer in "stop and > fsck" policing of disk drives. Me too. Though with a 32TB drive (I'm guessing rotating media), that's going to take a long time. If I had a drive that big, I'd divide it into managable chunks and mount them all under /drive/{a,b,c,d,e...} so that when something goes wrong you don't have to check the whole 32TB. > Near the top of the manual page it says: > > Warning > Do not use --repair unless you are advised to do so by a developer > or an experienced user, and then only after having accepted that > no fsck successfully repair all types of filesystem corruption. Eg. > some other software or hardware bugs can fatally damage a volume. > > Whoa! I'm sure that operators are standing by, call 1-800-FIX-BTRFS. > Really? Is a ploy by the developers to form a support business? That's a stretch, they are just trying to not encourage you to make a mess. I sent Linus an email to find out where btrfs is, I'll report back when he replies. --lm From robpike at gmail.com Mon Aug 30 11:21:29 2021 From: robpike at gmail.com (Rob Pike) Date: Mon, 30 Aug 2021 11:21:29 +1000 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <20210829235745.GC20021@mcvoy.com> References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> <20210829235745.GC20021@mcvoy.com> Message-ID: Do you even have input switches and HALT/CONT switches? I don't think so.... Commiserations. -rob On Mon, Aug 30, 2021 at 9:58 AM Larry McVoy wrote: > On Sun, Aug 29, 2021 at 03:12:16PM -0700, Jon Steinhart wrote: > > After a bit of poking around I discovered that btrfs SILENTLY remounted > the > > filesystem because it had errors. Sure, it put something in a log file, > > but I don't spend all day surfing logs for things that shouldn't be going > > wrong. Maybe my expectation that filesystems just work is antiquated. > > I give them credit for remounting read-only when seeing errors, they may > have gotten that from BitKeeper. When we opened a history file, if we > encountered any errors we opened the history file in read only mode so > if it worked enough you could see your data, great, but don't write on > top of bad data. > > > Although it's been discredited by some, I'm still a believer in "stop and > > fsck" policing of disk drives. > > Me too. Though with a 32TB drive (I'm guessing rotating media), that's > going to take a long time. If I had a drive that big, I'd divide it > into managable chunks and mount them all under /drive/{a,b,c,d,e...} > so that when something goes wrong you don't have to check the whole > 32TB. > > > Near the top of the manual page it says: > > > > Warning > > Do not use --repair unless you are advised to do so by a developer > > or an experienced user, and then only after having accepted that > > no fsck successfully repair all types of filesystem corruption. Eg. > > some other software or hardware bugs can fatally damage a volume. > > > > Whoa! I'm sure that operators are standing by, call 1-800-FIX-BTRFS. > > Really? Is a ploy by the developers to form a support business? > > That's a stretch, they are just trying to not encourage you to make a > mess. > > I sent Linus an email to find out where btrfs is, I'll report back when > he replies. > > --lm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tytso at mit.edu Mon Aug 30 13:14:28 2021 From: tytso at mit.edu (Theodore Ts'o) Date: Sun, 29 Aug 2021 23:14:28 -0400 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: On Sun, Aug 29, 2021 at 07:09:50PM -0400, Henry Bent wrote: > On Sun, 29 Aug 2021 at 18:13, Jon Steinhart wrote: > > > I recently upgraded my machines to fc34. I just did a stock > > uncomplicated installation using the defaults and it failed miserably. > > > > Fc34 uses btrfs as the default filesystem so I thought that I'd give it > > a try. > > ... cut out a lot about how no sane person would want to use btrfs ... The ext2/ext3/ext4 file system utilities is as far as I know the first fsck that was developed with a full regression test suite from the very beginning and integrated into the sources. (Just run "make check" and you'll know if you broken something --- or it's how I know the person contributing code was sloppy and didn't bother to run "make check" before sending me patches to review....) What a lot of people don't seem to understand is that file system utilities are *important*, and more work than you might think. The ext4 file system is roughly 71 kLOC (thousand lines of code) in the kernel. E2fsprogs is 340 kLOC. In contrast, the btrfs kernel code is 145 kLOC (btrfs does have a lot more "sexy new features"), but its btrfs-progs utilities is currently only 124 kLOC. And the e2fsprogs line count doesn't include the 350+ library of corrupted file system images that are part of its regression test suite. Btrfs has a few unit tests (as does e2fsprogs), but it doesn't have any thing similar in terms of a library corrupted file system images to test its fsck functionality. (Then again, neither does the file system utilities for FFS, so a regression test suite is not required to create a high quality fsck program. In my opinion, it very much helps, though!) > > Or, as Saturday Night Live might put it: And now, linux, starring the > > not ready for prime time filesystem. Seems like something that's been > > under development for around 15 years should be in better shape. > > > > To my way of thinking this isn't a Linux problem, or even a btrfs problem, > it's a Fedora problem. They're the ones who decided to switch their > default filesystem to something that clearly isn't ready for prime time. I was present at the very beginning of btrfs. In November, 2007, various file system developers from a number of the big IBM companies got together (IBM, Intel, HP, Red Hat, etc.) and folks decided that Linux "needed an answer to ZFS". In preparation for that meeting, I did some research asking various contacts I had at various companies how much effort and how long it took to create a new file system from scratch and make it be "enterprise ready". I asked folks at Digital how long it took for advfs, IBM for AIX and GPFS, etc., etc. And the answer I got back at that time was between 50 and 200 Person Years, with the bulk of the answers being between 100-200 PY's (the single 50PY estimate was an outlier). This was everything --- kernel and userspace coding, testing and QA, performance tuning, documentation, etc. etc. The calendar-time estimates I was given was between 5-7 calendar years, and even then, users would take at least another 2-3 years minimum of "kicking the tires", before they would trust *their* precious enterprise data on the file system. There was an Intel engineer at that meeting, who shall remain nameless, who said, "Don't tell the managers that or they'll never greenlight the project! Tell them 18 months...." And so I and other developers at IBM, continued working on ext4, which we never expected would be able to compete with btrfs and ZFS in terms of "sexy new features", but our focus was on performance, scalability, and robustness. And it probably was about 2015 or so that btrfs finally became more or less stable, but only if you restricted yourself to core functionality. (e.g., snapshots, file-system level RAID, etc., was still dodgy at the time.) I will say that at Google, ext4 is still our primary file system, mainly because all of our expertise is currently focused there. We are starting to support XFS in "beta" ("Preview") for Cloud Optimized OS, since there are some enterprise customers which are using XFS on their systems, and they want to continue using XFS as they migrate from on-prem to the Cloud. We fully support XFS for Anthos Migrate (which is a read-mostly workload), and we're still building our expertise, working on getting bug fixes backported, etc., so we can support XFS the way enterprises expect for Cloud Optimized OS, which is our high-security, ChromeOS based Linux distribution with a read-only, cryptographically signed root file system optimized for Docker and Kubernetes workloads. I'm not aware of any significant enterprise usage of btrfs, which is why we're not bothering to support btrfs at $WORK. The only big company which is using btrfs in production that I know of is Facebook, because they have a bunch of btrfs developers, but even there, they aren't using btrfs exclusively for all of their workloads. My understanding of why Fedora decided to make btrfs the default was because they wanted to get more guinea pigs to flush out the bugs. Note that Red Hat, which is responsible for Red Hat Enterprise Linux (their paid product, where they make $$$) and Fedora, which is their freebie "community distribution" --- Well, Red Hat does not currently support btrfs for their RHEL product. Make of that what you will.... - Ted From bakul at iitbombay.org Mon Aug 30 13:36:37 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Sun, 29 Aug 2021 20:36:37 -0700 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: Chances are your disk has a URE 1 in 10^14 bits ("enterprise" disks may have a URE of 1 in 10^15). 10^14 bit is about 12.5TB. For 16TB disks you should use at least mirroring, provided some day you'd want to fill up the disk. And a machine with ECC RAM (& trust but verify!). I am no fan of btrfs but these are the things I'd consider for any FS. Even if you have done all this, consider the fact that disk mortality has a bathtub curve. I use FreeBSD + ZFS so I'd recommend ZFS (on Linux). ZFS scrub works in background on an active system. Similarly resilvering (thought things slow down). On my original zfs filesystem I replaced all the 4 disks twice. I have been using zfs since 2005 and it has rarely required any babysitting. I reboot it when upgrading to a new release or applying kernel patches. "backups" via zfs send/recv of snapshots. > On Aug 29, 2021, at 3:12 PM, Jon Steinhart wrote: > > I recently upgraded my machines to fc34. I just did a stock > uncomplicated installation using the defaults and it failed miserably. > > Fc34 uses btrfs as the default filesystem so I thought that I'd give it > a try. I was especially interested in the automatic checksumming because > the majority of my storage is large media files and I worry about bit > rot in seldom used files. I have been keeping a separate database of > file hashes and in theory btrfs would make that automatic and transparent. > > I have 32T of disk on my system, so it took a long time to convert > everything over. A few weeks after I did this I went to unload my > camera and couldn't because the filesystem that holds my photos was > mounted read-only. WTF? I didn't do that. > > After a bit of poking around I discovered that btrfs SILENTLY remounted the > filesystem because it had errors. Sure, it put something in a log file, > but I don't spend all day surfing logs for things that shouldn't be going > wrong. Maybe my expectation that filesystems just work is antiquated. > > This was on a brand new 16T drive, so I didn't think that it was worth > the month that it would take to run the badblocks program which doesn't > really scale to modern disk sizes. Besides, SMART said that it was fine. > > Although it's been discredited by some, I'm still a believer in "stop and > fsck" policing of disk drives. Unmounted the filesystem and ran fsck to > discover that btrfs had to do its own thing. No idea why; I guess some > think that incompatibility is a good thing. > > Ran "btrfs check" which reported errors in the filesystem but was otherwise > useless BECAUSE IT DIDN'T FIX ANYTHING. What good is knowing that the > filesystem has errors if you can't fix them? > > Near the top of the manual page it says: > > Warning > Do not use --repair unless you are advised to do so by a developer > or an experienced user, and then only after having accepted that > no fsck successfully repair all types of filesystem corruption. Eg. > some other software or hardware bugs can fatally damage a volume. > > Whoa! I'm sure that operators are standing by, call 1-800-FIX-BTRFS. > Really? Is a ploy by the developers to form a support business? > > Later on, the manual page says: > > DANGEROUS OPTIONS > --repair > enable the repair mode and attempt to fix problems where possible > > Note there’s a warning and 10 second delay when this option > is run without --force to give users a chance to think twice > before running repair, the warnings in documentation have > shown to be insufficient > > Since when is it dangerous to repair a filesystem? That's a new one to me. > > Having no option other than not being able to use the disk, I ran btrfs > check with the --repair option. It crashed. Lesson so far is that > trusting my data to an unreliable unrepairable filesystem is not a good > idea. Since this was one of my media disks I just rebuilt it using ext4. > > Last week I was working away and tried to write out a file to discover > that /home and /root had become read-only. Charming. Tried rebooting, > but couldn't since btrfs filesystems aren't checked and repaired. Plugged > in a flash drive with a live version, managed to successfully run --repair, > and rebooted. Lasted about 15 minutes before flipping back to read only > with the same error. > > Time to suck it up and revert. Started a clean reinstall. Got stuck > because it crashed during disk setup with anaconda giving me a completely > useless big python stack trace. Eventually figured out that it was > unable to delete the btrfs filesystem that had errors so it just crashed > instead. Wiped it using dd; nice that some reliable tools still survive. > Finished the installation and am back up and running. > > Any of the rest of you have any experiences with btrfs? I'm sure that it > works fine at large companies that can afford a team of disk babysitters. > What benefits does btrfs provide that other filesystem formats such as > ext4 and ZFS don't? Is it just a continuation of the "we have to do > everything ourselves and under no circumstances use anything that came > from the BSD world" mentality? > > So what's the future for filesystem repair? Does it look like the past? > Is Ken's original need for dsw going to rise from the dead? > > In my limited experience btrfs is a BiTteR FileSystem to swallow. > > Or, as Saturday Night Live might put it: And now, linux, starring the > not ready for prime time filesystem. Seems like something that's been > under development for around 15 years should be in better shape. > > Jon -- Bakul From tytso at mit.edu Mon Aug 30 13:46:47 2021 From: tytso at mit.edu (Theodore Ts'o) Date: Sun, 29 Aug 2021 23:46:47 -0400 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <20210829235745.GC20021@mcvoy.com> References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> <20210829235745.GC20021@mcvoy.com> Message-ID: On Sun, Aug 29, 2021 at 04:57:45PM -0700, Larry McVoy wrote: > > I give them credit for remounting read-only when seeing errors, they may > have gotten that from BitKeeper. Actually, the btrfs folks got that from ext2/ext3/ext4. The original behavior was "don't worry, be happy" (log errors and continue), and I added two additional options, "remount read-only", and "panic and reboot the system". I recommend the last especially for high availability systems, since you can then fail over to the secondary system, and fsck can repair the file system on the reboot path. The primary general-purpose file systems in Linux which are under active development these days are btrfs, ext4, f2fs, and xfs. They all have slightly different focus areas. For example, f2fs is best for low-end flash, the kind that is find on $30 dollar mobile handsets on sale in countries like India (aka, "the next billion users"). It has deep knowledge of "cost-optimized" flash where random writes are to be avoided at all costs because write amplification is a terrible thing with very primitive FTL's. For very large file systems (e.g., large RAID arrays with pedabytes of data), XFS will probably do better than ext4 for many workloads. Btrfs is the file systems for users who have ZFS envy. I believe many of those sexy new features are best done at other layers in the storage stack, but if you *really* want file-system level snapshots and rollback, btrfs is the only game in town for Linux. (Unless of course you don't mind using ZFS and hope that Larry Ellison won't sue the bejesus out of you, and if you don't care about potential GPL violations....) Ext4 is still getting new features added; we recently added a light-weight journaling (a simplified version of the 2017 Usenix ATC iJournaling paper[1]), and just last week we've added parallelized orphan list called Orphan File[2] which optimizes parallel truncate and unlink workloads. (Neither of these features are enabled by default yet, because maybe in a few years, or earlier if community distros want to volunteer their users to be guinea pigs. :-) [1] https://www.usenix.org/system/files/conference/atc17/atc17-park.pdf [2] https://www.spinics.net/lists/linux-ext4/msg79021.html We currently aren't adding the "sexy new features" of btrfs or ZFS, but that's mainly because there isn't a business justification to pay for the engineering effort needed to add them. I have some design sketches of how we *could* add them to ext4, but most of the ext4 developers like food with our meals, and I'm still a working stiff so I focus on work that adds value to my employer --- and, of course, helping other ext4 developers working at other companies figure out ways to justify new features that would add value to *their* employers. I might work on some sexy new features if I won the Powerball Lottery and could retire rich, or I was working at company where engineers could work on whatever technologies they wanted without getting permission from the business types, but those companies tend not to end well (especially after they get purchased by Oracle....) - Ted From johnmdow at me.com Mon Aug 30 19:14:15 2021 From: johnmdow at me.com (John Dow) Date: Mon, 30 Aug 2021 10:14:15 +0100 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: <1e976247-3642-8f84-a429-167e8b640877@me.com> On 30/08/2021 00:09, Henry Bent wrote: > On Sun, 29 Aug 2021 at 18:13, Jon Steinhart > wrote: > > I recently upgraded my machines to fc34.  I just did a stock > uncomplicated installation using the defaults and it failed miserably. > > Fc34 uses btrfs as the default filesystem so I thought that I'd > give it > a try. > > > ... cut out a lot about how no sane person would want to use btrfs ... > > > Or, as Saturday Night Live might put it:  And now, linux, starring the > not ready for prime time filesystem.  Seems like something that's been > under development for around 15 years should be in better shape. > > > To my way of thinking this isn't a Linux problem, or even a btrfs > problem, it's a Fedora problem.  They're the ones who decided to > switch their default filesystem to something that clearly isn't ready > for prime time. This. Even the Arch wiki makes it clear that btrfs isn't ready for prime time. It's still under very heavy development - not really something you want for a filesystem, particularly one storing a large amount of critical data. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From tytso at mit.edu Mon Aug 30 21:56:39 2021 From: tytso at mit.edu (Theodore Ts'o) Date: Mon, 30 Aug 2021 07:56:39 -0400 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: On Sun, Aug 29, 2021 at 08:36:37PM -0700, Bakul Shah wrote: > Chances are your disk has a URE 1 in 10^14 bits ("enterprise" disks > may have a URE of 1 in 10^15). 10^14 bit is about 12.5TB. For 16TB > disks you should use at least mirroring, provided some day you'd want > to fill up the disk. And a machine with ECC RAM (& trust but verify!). > I am no fan of btrfs but these are the things I'd consider for any FS. > Even if you have done all this, consider the fact that disk mortality > has a bathtub curve. You may find this article interesting: "The case of the 12TB URE: Explained and debunked"[1], and the following commit on a reddit post[2] discussiong this article: "Lol of course it's a myth. I don't know why or how anyone thought there would be a URE anywhere close to every 12TB read. Many of us have large pools that are dozens and sometimes hundreds of TB. I have 2 64TB pools and scrub them every month. I can go years without a checksum error during a scrub, which means that all my ~50TB of data was read correctly without any URE many times in a row which means that I have sometimes read 1PB (50TB x 2 pools x 10 months) worth from my disks without any URE. Last I checked, the spec sheets say < 1 in 1x1014 which means less than 1 in 12TB. 0 in 1PB is less than 1 in 12TB so it meets the spec." [1] https://heremystuff.wordpress.com/2020/08/25/the-case-of-the-12tb-ure/ [2] https://www.reddit.com/r/DataHoarder/comments/igmab7/the_12tb_ure_myth_explained_and_debunked/ Of course, disks do die, and ECC and backups and checksums are good things. But the whole "read 12TB get an error", saying really misunderstands how hdd failures work. Losing an entire platter, or maybe the entire 12TB disk die due to a head crash, adds a lot of uncorrectable read errors to the numerator of the UER statistics. It just goes to show that human intuition really sucks at statistics, whether it's about vaccination side effects, nuclear power plants, or the danger of flying in airplanes versus driving in cars. - Ted From norman at oclsc.org Mon Aug 30 23:06:03 2021 From: norman at oclsc.org (Norman Wilson) Date: Mon, 30 Aug 2021 09:06:03 -0400 (EDT) Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? Message-ID: <20210830130603.A7D4C640CC6@lignose.oclsc.org> Not to get into what is soemthing of a religious war, but this was the paper that convinced me that silent data corruption in storage is worth thinking about: http://www.cs.toronto.edu/~bianca/papers/fast08.pdf A key point is that the character of the errors they found suggests it's not just the disks one ought to worry about, but all the hardware and software (much of the latter inside disks and storage controllers and the like) in the storage stack. I had heard anecdotes long before (e.g. from Andrew Hume) suggesting silent data corruption had become prominent enough to matter, but this paper was the first real study I came across. I have used ZFS for my home file server for more than a decade; presently on an antique version of Solaris, but I hope to migrate to OpenZFS on a newer OS and hardware. So far as I can tell ZFS in old Solaris is quite stable and reliable. As Ted has said, there are philosophical reasons why some prefer to avoid it, but if you don't subscribe to those it's a fine answer. I've been hearing anecdotes since forever about sharp edges lurking here and there in BtrFS. It does seem to be eternally unready for production use if you really care about your data. It's all anecdotes so I don't know how seriously to take it, but since I'm comfortable with ZFS I don't worry about it. Norman Wilson Toronto ON PS: Disclosure: I work in the same (large) CS department as Bianca Schroeder, and admire her work in general, though the paper cited above was my first taste of it. From tytso at mit.edu Tue Aug 31 00:42:53 2021 From: tytso at mit.edu (Theodore Ts'o) Date: Mon, 30 Aug 2021 10:42:53 -0400 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <20210830130603.A7D4C640CC6@lignose.oclsc.org> References: <20210830130603.A7D4C640CC6@lignose.oclsc.org> Message-ID: On Mon, Aug 30, 2021 at 09:06:03AM -0400, Norman Wilson wrote: > Not to get into what is soemthing of a religious war, > but this was the paper that convinced me that silent > data corruption in storage is worth thinking about: > > http://www.cs.toronto.edu/~bianca/papers/fast08.pdf > > A key point is that the character of the errors they > found suggests it's not just the disks one ought to worry > about, but all the hardware and software (much of the latter > inside disks and storage controllers and the like) in the > storage stack. There's nothing I'd disagree with in this paper. I'll note though that part of the paper's findings is that silent data corruption occured at a rate roughly 1.5 orders of magnitude less often than latent sector errors (e.g., UER's). The probability of at least one silent data corruption during the study period (41 months, although not all disks would have been in service during that entire time) was P = 0.0086 for nearline disks and P = 0.00065 for enterprise disks. And P(1st error) remained constant over disk age, and per the authors' analysis, it was unclear whether P(1st error) changed as disk size increased --- which is representative of non media-related failures (not surprising given how much checksum and ECC checks are done by the HDD's). A likely supposition IMHO is that the use of more costly enterprise disks correlated with higher quality hardware in the rest of the storage stack --- so things like ECC memory really do matter. > As Ted has said, there are philosophical reasons why some prefer to > avoid it, but if you don't subscribe to those it's a fine answer. WRT to running ZFS on Linux, I wouldn't call it philosophical reasons, but rather legal risks. Life is not perfect, so you can't drive any kind of risk (including risks of hardware failure) down to zero. Whether you should be comfortable with the legal risks in this case very much depends on who you are and what your risk profile might be, and you should contact a lawyer if you want legal advice. Clearly the lawyers at companies like Red Hat and SuSE have given very answers from the lawyers at Canonical. In addition, the answer for hobbyists and academics might be quite different from a large company making lots of money and more likely to attract the attention of the leadership and lawyers at Oracle. Cheers, - Ted From krewat at kilonet.net Tue Aug 31 02:46:59 2021 From: krewat at kilonet.net (Arthur Krewat) Date: Mon, 30 Aug 2021 12:46:59 -0400 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <20210830130603.A7D4C640CC6@lignose.oclsc.org> References: <20210830130603.A7D4C640CC6@lignose.oclsc.org> Message-ID: <434d8b8d-ed64-cc73-9947-3d415a90bb08@kilonet.net> On 8/30/2021 9:06 AM, Norman Wilson wrote: > A key point is that the character of the errors they > found suggests it's not just the disks one ought to worry > about, but all the hardware and software (much of the latter > inside disks and storage controllers and the like) in the > storage stack. I had a pair of Dell MD1000's, full of SATA drives (28 total), with the SATA/SAS interposers on the back of the drive. Was getting checksum errors in ZFS on a handful of the drives. Took the time to build a new array, on a Supermicro backplane, and no more errors with the exact same drives. I'm theorizing it was either the interposers, or the SAS backplane/controllers in the MD1000. Without ZFS, who knows who swiss-cheesy my data would be. Not to mention the time I setup a Solaris x86 cluster zoned to a Compellent and periodically would get one or two checksum errors in ZFS. This was the only cluster out of a handful that had issues, and only on that one filesystem. Of course, it was a production PeopleSoft Oracle database. I guess moving to a VMware Linux guest and XFS just swept the problem under the rug, but the hardware is not being reused so there's that. > I had heard anecdotes long before (e.g. from Andrew Hume) > suggesting silent data corruption had become prominent > enough to matter, but this paper was the first real study > I came across. > > I have used ZFS for my home file server for more than a > decade; presently on an antique version of Solaris, but > I hope to migrate to OpenZFS on a newer OS and hardware. > So far as I can tell ZFS in old Solaris is quite stable > and reliable. As Ted has said, there are philosophical > reasons why some prefer to avoid it, but if you don't > subscribe to those it's a fine answer. > Been running Solaris 11.3 and ZFS for quite a few years now, at home. Before that, Solaris 10. I recently setup a home Redhat 8 server, w/ZoL (.8), earlier this year - so far, no issues, with 40+TB online. I have various test servers with ZoL 2.0 on them, too. I have so much online data that I use as the "live copy" - going back to the early 80's copies of my TOPS-10 stuff. Even though I have copious amounts of LTO tape copies of this data, I won't go back to the "out of sight out of mind" mentality. Trying to get customers to buy into that idea is another story. art k. PS: I refuse to use a workstation that doesn't use ECC RAM, either. I like swiss-cheese on a sandwich. I don't like my (or my customers') data emulating it. From steffen at sdaoden.eu Mon Aug 30 23:55:08 2021 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Mon, 30 Aug 2021 15:55:08 +0200 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: <20210830135508.oKucm%steffen@sdaoden.eu> Theodore Ts'o wrote in : ... |What a lot of people don't seem to understand is that file system |utilities are *important*, and more work than you might think. The |ext4 file system is roughly 71 kLOC (thousand lines of code) in the |kernel. E2fsprogs is 340 kLOC. In contrast, the btrfs kernel code is |145 kLOC (btrfs does have a lot more "sexy new features"), but its |btrfs-progs utilities is currently only 124 kLOC. To be "fair" it must be said that btrfs usage almost requires installation of e2fsprogs because only that ships chattr(1). --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From steffen at sdaoden.eu Tue Aug 31 01:05:10 2021 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Mon, 30 Aug 2021 17:05:10 +0200 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: <20210830150510.jWrRZ%steffen@sdaoden.eu> Jon Steinhart wrote in <202108292212.17TMCGow1448973 at darkstar.fourwinds.com>: |I recently upgraded my machines to fc34. I just did a stock |uncomplicated installation using the defaults and it failed miserably. | |Fc34 uses btrfs as the default filesystem so I thought that I'd give it ... |Any of the rest of you have any experiences with btrfs? I'm sure that it |works fine at large companies that can afford a team of disk babysitters. |What benefits does btrfs provide that other filesystem formats such as |ext4 and ZFS don't? Is it just a continuation of the "we have to do |everything ourselves and under no circumstances use anything that came |from the BSD world" mentality? From a small man perspective i can say i use btrfs. I have seen more problems with filesystems in these 29 months than in the ~36 years (well, 22 if you only count Unix, mostly FreeBSD) before. I have learned that i have to chattr +C my vm/ directory in order to avoid filesystem errors which can only be solved by deleting the corrupted files (which were not easy to find out, inspect-internal inode-resolve could have helped a bit better, but .. why?). I have seen (factual) snapshot receive errors that were not indicated by the tool's exit status. With dangling files laying around. I have seen +C attributes missing on the target filesystem from played in snapshots. I have found it impossible to mount external devices because "BTRFS warning (device ): duplicate device /dev/sdc2 devid 1 generation 3977" even though the former umount was successful (i must admit i had used lazytime mount option there, be sure not to do that for BTRFS). I know where you coming from, i tried it all without success. With all the little tools that do not do what you want. I found it quite ridiculous that i had to shrink-resize my filesystem in an iteration because the "minimum possible" size was adjusted each and every time, until i finally was where the actual filesystem usage indicated you would end up with from the beginning. (defrag did not prevent this.) On the other hand i am still here, now on Luks2, now with xxhash checksums and data and meta DUP (instead of RAID 1, i thought). Same data, just moved over onto. I think i would try out ZFS if it would be in-kernel. The FreeBSD Handbook is possibly the best manual you can have for that. But i mean the ZFS code is much, much larger than BTRFS, BTRFS serves me really, really good for what i want and need, and i am sailing Linux stable/latest (4.19, now 5.10) here, on a free and small Linux distribution without engineering power aka supervision, that sails somewhat the edge of what all the many involved projects produce, with problems all over the place, sometimes more, sometimes less, some all the time. That is nothing new, you have to live with some deficiencies in the free software world; i just find it sometimes hard to believe that this is still true for Linux with the many many billions of Dollars that went in, and the tens of thousands of people working on it. I really hate that all the Linux kernel guys seem to look forward only, mostly, you have to go and try out the newest thing, maybe there all is good. Of course .. i can understand this (a bit). That is the good thing of using such an engineering-power distribution, you likely get backports and have a large user base with feedback. |In my limited experience btrfs is a BiTteR FileSystem to swallow. Well often people say you need to know what you are doing. That is hard without proper documentation, and the www is a toilet. And noone writes HOWTOs any more. But, for example, if i do not use an undocumented kernel parameter (rtw88_pci.disable_aspm=1), and use wifi and bluetooth (audio) in conjunction, then i have to boot into the Windows startup screen in order to properly reinitialize my wifi/bluetooth chip. Or else you are dead. And that even though it seems the Linux driver comes from the Chip creator itself. So i think a "chattr +C" here and there, it can be directory-wide, it could be a mount or creation option, isn't that bad. Also it is just me having had a go (or julia or nim; or perl) on using _one_ filesystem with several subvolumes for anything; if i would have used my (well, security context only) old-style way of doing things and used several hard partitions with individual filesystems, i would have used +C from the beginning for that one. Especially if i would have read it in the manual page. I do believe todays' journalled, checksummed, snapshot'able copy-on-write filesystems are complex beasts. --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From athornton at gmail.com Tue Aug 31 04:08:22 2021 From: athornton at gmail.com (Adam Thornton) Date: Mon, 30 Aug 2021 11:08:22 -0700 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: References: <20210830130603.A7D4C640CC6@lignose.oclsc.org> Message-ID: <561678CC-6C29-43E7-AA37-1F3FAE5AE94A@gmail.com> > On Aug 30, 2021, at 7:42 AM, Theodore Ts'o wrote: > In addition, the answer for hobbyists > and academics might be quite different from a large company making > lots of money and more likely to attract the attention of the > leadership and lawyers at Oracle. I mean, Oracle’s always been super well-behaved and not lawsuit-happy _at all_! You’re fine, at least until Larry needs a slightly larger yacht or volcano lair or whatever it is he’s into these days. Adam From lm at mcvoy.com Tue Aug 31 07:38:41 2021 From: lm at mcvoy.com (Larry McVoy) Date: Mon, 30 Aug 2021 14:38:41 -0700 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: <20210830213841.GA13471@mcvoy.com> Linus replied: > Anyhoo, what can you me about btrfs? I haven't paid attention, when I > last did, it seemed unfinished. Fedora apparently uses it by default, > should they be? I've been pretty happy with ext{2,3,4}, I don't know > if Ted is still behind those but I've been super happy with how they > handled compat and other stuff. > > So did btrfs get done? I'm not sure how much people use the fancier features, but lots of people seem to use btrfs in production these days, and the maintenance team seems responsive and stable. Of the local disk filesystems we have, I'd pick any of ext4, btrfs and xfs as stable and well-maintained. Picking any of those is fine, and the choice would come down to which distro you use, and _possibly_ whether you want any of the particular features. But on the whole, most people don't seem to care deeply about the more intricate features unless they have very specific use-cases. From bakul at iitbombay.org Tue Aug 31 08:35:44 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Mon, 30 Aug 2021 15:35:44 -0700 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> Message-ID: <7E649324-2690-4D9D-A175-B74089EFEE9A@iitbombay.org> On Aug 30, 2021, at 4:56 AM, Theodore Ts'o wrote: > > On Sun, Aug 29, 2021 at 08:36:37PM -0700, Bakul Shah wrote: >> Chances are your disk has a URE 1 in 10^14 bits ("enterprise" disks >> may have a URE of 1 in 10^15). 10^14 bit is about 12.5TB. For 16TB >> disks you should use at least mirroring, provided some day you'd want >> to fill up the disk. And a machine with ECC RAM (& trust but verify!). >> I am no fan of btrfs but these are the things I'd consider for any FS. >> Even if you have done all this, consider the fact that disk mortality >> has a bathtub curve. > > You may find this article interesting: "The case of the 12TB URE: > Explained and debunked"[1], and the following commit on a reddit > post[2] discussiong this article: > > "Lol of course it's a myth. > > I don't know why or how anyone thought there would be a URE > anywhere close to every 12TB read. > > Many of us have large pools that are dozens and sometimes hundreds of TB. > > I have 2 64TB pools and scrub them every month. I can go years > without a checksum error during a scrub, which means that all my > ~50TB of data was read correctly without any URE many times in a > row which means that I have sometimes read 1PB (50TB x 2 pools x 10 > months) worth from my disks without any URE. > > Last I checked, the spec sheets say < 1 in 1x1014 which means less > than 1 in 12TB. 0 in 1PB is less than 1 in 12TB so it meets the > spec." > > [1] https://heremystuff.wordpress.com/2020/08/25/the-case-of-the-12tb-ure/ > [2] https://www.reddit.com/r/DataHoarder/comments/igmab7/the_12tb_ure_myth_explained_and_debunked/ It seems this guy doesn't understand statistics. He checked his 2 pools and is confident that a sample of 4 disks (likely) he knows that URE specs are crap. Even from an economic PoV it doen't make sense. Why wouldn't the disk companies tout an even lower error rate if they can get away with it? Presumably these rates are derived from reading many many disks and averaged. Here's what the author says on a serverfault thread: https://serverfault.com/questions/812891/what-is-exactly-an-ure @DavidBalažic Evidently, your sample size of one invalidates the entirety of probability theory! I suggest you submit a paper to the Nobel Committee. – Ian Kemp Apr 16 '19 at 5:37 @IanKemp If someone claims that all numbers are divisible by 7 and I find ONE that is not, then yes, a single find can invalidate an entire theory. BTW, still not a single person has confirmed the myth in practice (by experiment), did they? Why should they, when belief is more than knowledge...– David Balažic Apr 16 '19 at 12:22 Incidentally, it is hard to believe he scrubs his 2x64TB pools once a month. Assuming 250MB/s sequential throughput and his scrubber can stream it at that rate, it will take him close to 6 days (3 days if reading them in parallel) to read every block. During this time these pools won't be useful for anything else. Unclear if he is using any RAID or a filesystem that does checksums. Without that he would be unable to detect hidden data corruption. In contrast, ZFS will only scrub *live* data. As more of the disks are filled up, scrub will take progressively more time. Similarly, replacing a zfs mirror won't read the source disk in its entirety, only the live data. > Of course, disks do die, and ECC and backups and checksums are good > things. But the whole "read 12TB get an error", saying really > misunderstands how hdd failures work. Losing an entire platter, or > maybe the entire 12TB disk die due to a head crash, adds a lot of > uncorrectable read errors to the numerator of the UER statistics. That is not how URE specs are derived. > > It just goes to show that human intuition really sucks at statistics, Indeed :-) From bakul at iitbombay.org Tue Aug 31 09:04:26 2021 From: bakul at iitbombay.org (Bakul Shah) Date: Mon, 30 Aug 2021 16:04:26 -0700 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> <20210829235745.GC20021@mcvoy.com> Message-ID: <71B14DD4-0F7D-4AE1-9BCE-3327C056FFD2@iitbombay.org> On Aug 29, 2021, at 8:46 PM, Theodore Ts'o wrote: > > (Unless of > course you don't mind using ZFS and hope that Larry Ellison won't sue > the bejesus out of you, and if you don't care about potential GPL > violations....) This should not matter if you are adding zfs for your own use. Still, if this is a concern, best switch over to FreeBSD :-) Actually FreeBSD is now using OpenZFS so it is the same source code base as ZFS on Linux. You should even be able to do zfs send/recv between the two OSes. From steffen at sdaoden.eu Tue Aug 31 23:18:46 2021 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Tue, 31 Aug 2021 15:18:46 +0200 Subject: [TUHS] Is it time to resurrect the original dsw (delete with switches)? In-Reply-To: <20210830150510.jWrRZ%steffen@sdaoden.eu> References: <202108292212.17TMCGow1448973@darkstar.fourwinds.com> <20210830150510.jWrRZ%steffen@sdaoden.eu> Message-ID: <20210831131846.v32D7%steffen@sdaoden.eu> P.S.: Steffen Nurpmeso wrote in <20210830150510.jWrRZ%steffen at sdaoden.eu>: |Jon Steinhart wrote in | <202108292212.17TMCGow1448973 at darkstar.fourwinds.com>: ||I recently upgraded my machines to fc34. I just did a stock ||uncomplicated installation using the defaults and it failed miserably. || ||Fc34 uses btrfs as the default filesystem so I thought that I'd give it ... |I have learned that i have to chattr +C my vm/ directory in order |to avoid filesystem errors which can only be solved by deleting |the corrupted files (which were not easy to find out, |inspect-internal inode-resolve could have helped a bit better, but ... |creator itself. So i think a "chattr +C" here and there, it can |be directory-wide, it could be a mount or creation option, isn't |that bad. Also it is just me having had a go (or julia or nim; or ... Only to add that this was effectively my fault, because of the caching behaviour my box-vm.sh script chose for qemu. In effect i think that +C i could drop again now that i have drivecache=,cache=writeback # ,cache=none XXX on ZFS!? used like eg -drive index=0,if=ide$drivecache,file=img/$vmimg It, however, took quite some research to get there. --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)