From neitzel@spog.gaertner.de Mon Jun 22 11:13:19 1998 Received: (from neitzel@localhost) by spog.gaertner.de (8.8.8/8.8.8/Nase) id LAA08751; Mon, 22 Jun 1998 11:13:19 +0200 Message-Id: <199806220913.LAA08751@spog.gaertner.de> Date: Mon, 22 Jun 1998 11:13:19 +0200 From: JKT@Hexagon.Com Reply-To: JKT@Hexagon.Com To: bugs@spog.gaertner.de Cc: kbi@interlog.com Subject: Using the "J shell" jrun as a login shell X-Send-Pr-Version: 3.104 >Number: 4 >Category: j-unix >Synopsis: Using the "J shell" jrun as a login shell >Confidential: no >Severity: serious >Priority: medium >Responsible: neitzel >State: closed >Class: support >Submitter-Id: hexagon >Arrival-Date: Mon Jun 22 11:22:01 MET DST 1998 >Last-Modified: Sat Feb 27 21:48:02 MET 1999 >Originator: Joey Tuttle >Organization: Hexagon >Release: >Environment: SunOS aptos-1 5.4 generic sun4m sparc unix in general >Description: My main wish is that I could get the J shell mechanism to work (in Unix V as well as BSD systems). I have had zero success. The examples described in Readme.Unix do not work in any envionment where I've tried them. I simply end up with a shell like sh where 'special characters' such as " and \ have to be escaped in order to use them. This makes writing J rather tedious and non-portable... I have tried various locations for jrun, and entering things in system tables - e.g. Sun Release 4.1 Last change: 6 October 1987 1 jkt@pocus$ cat /etc/shells /bin/sh /bin/csh /usr/local/bin/tcsh /usr/local/bin/jrun jkt@pocus$ All to no avail... If I could get jrun to really be a shell, I think the externals described (2!:X) would be satisfactory - but I haven't gotten far enough to know that for certain. Please let me know what I might/should do to get better results. >How-To-Repeat: >Fix: >Audit-Trail: From: Martin Neitzel <neitzel@gaertner.de> To: JKT@Hexagon.Com Cc: bugs@gaertner.de, kbi@interlog.com Subject: Re: j-unix/4: Using the "J shell" jrun as a login shell Date: Mon, 22 Jun 1998 11:57:16 +0200 (MET DST) "jrun" is a bit misleadingly described in the Readme.Unix as a "shell interpreter" but it is not intended as a login shell. The "jrun" interface to the "jint + certain profiles" is intended to have J scripts mentioning their interpreter with the "#!" magic cookie. The documentation is correct in this respect. With respect to the Readme.Unix as of J3.03: - The heading "Shell Interpreter" and the term "shell script" are at least very misleading terminology and should be revised. - The "reverse" example lacks the initial "#!" in line 1. However, I see your question about really getting a J environment as a login shell. I'll have a look into it. Martin From: "Joey K. Tuttle" <jkt@pocus.hexagon.com> To: Martin Neitzel <neitzel@gaertner.de> Cc: bugs@gaertner.de, kbi@interlog.com Subject: Re: j-unix/4: Using the "J shell" jrun as a login shell Date: Mon, 22 Jun 1998 12:21:50 -0700 At 11:57 +0200 98/6/22, Martin Neitzel wrote: >"jrun" is a bit misleadingly described in the Readme.Unix as a >"shell interpreter" but it is not intended as a login shell. > I'm really not interested in using J as a login shell... The documentation that I have says: " A shell script is a shortcut notation in Unix to allow a simple ASCII file to contain both the name of the shell interpreter and the sequence of instructions for that interpreter. The ASCII file (once flagged as executable with chmod) can be used just as any other executable. Examples of shell interpreters are the command processors like sh, csh, bsh, and ksh, and other useful interpreters such as awk and perl. The J interpreter, when invoked with the jrun script can also be used as a shell interpreter. The J instructions to run are included in the file, and facilities exist within J to determine how the script was called and to read and write standard input/output. A short example of a script which computes the total of the numbers in the command line follows: total--------------------+ | #! /usr/local/bin/jrun | | echo +/ > 0".&> }.ARGV | +------------------------+ I am looking for the behavior described in the first paragraph, especially the last sentence. I didn't find the description misleading in the sense that I didn't imagine using J as a login shell (I suppose that could be amusing...) My problem is that I cannot make the simple example script "total" work in any way that is vaguely similar to awk, perl, or other shell scripts... If you can tell me what I am doing wrong (or failiing to do) to get that script to work, I would be a happy camper. (the total example contains several of the characters that upset normal Unix shells (e.g. " * } and so on). >The "jrun" interface to the "jint + certain profiles" is intended >to have J scripts mentioning their interpreter with the "#!" magic >cookie. The documentation is correct in this respect. > I tried this, and assumed that the ASCII text for total should look as follows - jkt@aptos-1$ ls -l total -rwxr-xr-x 1 jkt faxfocus 47 Jun 22 12:07 total* jkt@aptos-1$ cat total #! /usr/local/bin/jrun echo +/ > 0".&> }.ARGV jkt@aptos-1$ cat /usr/local/bin/jrun #! /bin/sh # Copyright (c) 1997 Iverson Software Inc. All rights reserved. # # Invoke J as a shell interpreter. # if [ -z "${JLIB}" ] then JLIB=/export/home/jkt/j304/lib;export JLIB fi /export/home/jkt/j304/jint /export/home/jkt/j304/lib/extras/config/profile.js /export/home/jkt/j304/lib/main/files.js /export/home/j kt/j304/lib/shell.js -- $* jkt@aptos-1$ total 1 2 3 total[2]: syntax error at line 2 : `"' unmatched jkt@aptos-1$ The environment above is Solaris, but the same thing happened in BSD and Unix V. What am I missing??? The/A problem is that jrun is not being invoked even though it is explicitly mentioned in the first line of the shell script in the way other shell interpreters normally are. >With respect to the Readme.Unix as of J3.03: > >- The heading "Shell Interpreter" and the term "shell script" > are at least very misleading terminology and should be revised. > >- The "reverse" example lacks the initial "#!" in line 1. > >However, I see your question about really getting a J environment >as a login shell. I'll have a look into it. > Again, not my question, although an interesting one. Thanks -- -- Joey From: Martin Neitzel <neitzel@gaertner.de> To: jkt@pocus.hexagon.com Cc: bugs@gaertner.de, kbi@interlog.com Subject: Re: j-unix/4: Using the "J shell" jrun as a login shell Date: Mon, 22 Jun 1998 23:16:24 +0200 (MET DST) > jkt@aptos-1$ cat total > #! /usr/local/bin/jrun > echo +/ > 0".&> }.ARGV > The output above looks like there is a space before the #! magic cookie. This must not be there. See execve(2) for other details of this line. When you invoke "total" as-is from your interactive shell, this is what happens: The shell forks a new process, which is a shell, too. This child shell does an execve(2) (or some of its variants) to get replaced by the "total" program. execve(2) takes a look at the first 2 bytes (the "magic cookie") to see if it is dealing with a statically linked executable, a writable-code executable, etc., or a script having the #! cookie and the name of the interpreter executable. In the last case, the kernel creates a new command/argument vector and tries to exec the interpreter (which has then actually to open the script/data file on its own.) In your case however -- if there's the initial space character --, the execve(2) doesn't find a valid magic cookie at all. So it just returns a failure code to the shell. Now your child shell (which didn't get replaced by the failed exec) jumps back in and acts as follows: "No valid executable? Lessee if *I* can digest the file as a script." The shell will be happy with line 1, since it is just a comment to it (doesn't matter if you are using sh/csh/ksh/... as your interactive shell -- they all take "#" as their comment character). However, the second line is syntactically not digestable anymore, and therefore you get this message from the shell: > jkt@aptos-1$ total 1 2 3 > total[2]: syntax error at line 2 : `"' unmatched Everything clear so far? This behaviour of the shell has historic reasons. The very first unix, when there just was the /bin/sh and no other, didn't have the "#!" magic in exec(2). Shell scripts didn't start with any special introductory line but directly with the first command. (You can find this straight-forward style of scripts in all "classic" UNIX intros, for example Kernighan/Pike's "Unix Programming Environment".) When the Berkeley folks came up with the second shell (Bill Joy's csh), it did just the same thing with failed exec()s. Naturally, it interpreted the script according to its csh-syntax. The big problem now was that csh-users couldn't invoke sh scripts anymore (only with "sh fooscript"), and sh users couldn't invoke csh scripts just by their name. People had actually to know how a program was implemented, which is of course a bad thing. So the first #! line was born. Actually, many early unix systems had just the shells extended to take care of the #! line. But nowadays, this is almost everywhere a feature of the kernel (by virtue of the exec(2) implementation). Martin From: "Joey K. Tuttle" <jkt@pocus.hexagon.com> To: Martin Neitzel <neitzel@gaertner.de> Cc: bugs@gaertner.de, kbi@interlog.com, jhui@interlog.com Subject: Re: j-unix/4: Using the "J shell" jrun as a login shell Date: Mon, 22 Jun 1998 19:08:23 -0700 At 23:16 +0200 98/6/22, Martin Neitzel wrote: >> jkt@aptos-1$ cat total >> #! /usr/local/bin/jrun >> echo +/ > 0".&> }.ARGV >> > >The output above looks like there is a space before the #! magic cookie. >This must not be there. See execve(2) for other details of this line. > Yes, you are right - actually I just threw this together today and made the typo you picked up. I actually gave up on the whole thing a few months ago and then was encouraged by Roger to take it up with you and Kirk. I know the rules and even a good bit of the history that you bring out below - this is not new to me. But it also doesn't help... Here is the same thing after removing the (admittedly incorrect) blank. jkt@aptos-1$ cat total #! /usr/local/bin/jrun echo +/ > 0".&> }.ARGV jkt@aptos-1$ total 1 2 3 total[2]: syntax error at line 2 : `"' unmatched jkt@aptos-1$ #This is the result that I have seen and the one I would like to change. jkt@aptos-1$ date Mon Jun 22 18:49:45 PDT 1998 jkt@aptos-1$ jrun |index error | SCRIPT_z_=:SCRIPT}.~('#!'-:2{.SCRIPT)*>:0{#;._2 SCRIPT_z_=:1!:1]0 {ARGV |[-7] NB. it takes a surprisingly long time (3 or 4 seconds) to get to this point... 2+2 4 nl 3 +-------+--------+----+------+------+-----+------+------+--------+-------+-----+---+-------+-----+------+-------+ |fappend|fappends|fdir|ferase|fexist|fread|freadr|freads|freplace|fselect|fsize|fss|fssrplc|fview|fwrite|fwrites| +-------+--------+----+------+------+-----+------+------+--------+-------+-----+---+-------+-----+------+-------+ fss 4 : 0 y=. boxopen y. size=. 1!:4 :: _1: @ boxopen if. size -: _1 do. return. end. bx=. # i.@# blk=. (#x.) >. 100000 <. size r=. i.pos=. 0 while. pos < size do. dat=. 1!:11 y,<pos,blk <. size-pos r=. r,pos+bx x. E. dat pos=. pos+blk+1-#x. end. r ) boxopen <^:(L. = 0:) NB. ^D jkt@aptos-1$ Further suggestions??? >When you invoke "total" as-is from your interactive shell, this is what >happens: > >The shell forks a new process, which is a shell, too. >This child shell does an execve(2) (or some of its variants) >to get replaced by the "total" program. > >execve(2) takes a look at the first 2 bytes (the "magic cookie") to see >if it is dealing with a statically linked executable, a writable-code >executable, etc., or a script having the #! cookie and the name of the >interpreter executable. In the last case, the kernel creates a new >command/argument vector and tries to exec the interpreter (which has >then actually to open the script/data file on its own.) > >In your case however -- if there's the initial space character --, >the execve(2) doesn't find a valid magic cookie at all. So it >just returns a failure code to the shell. > >Now your child shell (which didn't get replaced by the failed exec) >jumps back in and acts as follows: >"No valid executable? Lessee if *I* can digest the file as a script." > >The shell will be happy with line 1, since it is just a comment to >it (doesn't matter if you are using sh/csh/ksh/... as your interactive >shell -- they all take "#" as their comment character). > >However, the second line is syntactically not digestable anymore, >and therefore you get this message from the shell: > >> jkt@aptos-1$ total 1 2 3 >> total[2]: syntax error at line 2 : `"' unmatched > >Everything clear so far? > >This behaviour of the shell has historic reasons. > >The very first unix, when there just was the /bin/sh and no other, >didn't have the "#!" magic in exec(2). Shell scripts didn't start with >any special introductory line but directly with the first command. >(You can find this straight-forward style of scripts in all "classic" >UNIX intros, for example Kernighan/Pike's "Unix Programming >Environment".) > >When the Berkeley folks came up with the second shell (Bill Joy's csh), >it did just the same thing with failed exec()s. Naturally, it >interpreted the script according to its csh-syntax. > >The big problem now was that csh-users couldn't invoke sh scripts >anymore (only with "sh fooscript"), and sh users couldn't invoke csh >scripts just by their name. People had actually to know how a program >was implemented, which is of course a bad thing. So the first #! line >was born. Actually, many early unix systems had just the shells >extended to take care of the #! line. But nowadays, this is almost >everywhere a feature of the kernel (by virtue of the exec(2) >implementation). > > Martin I would still like to run a J shell.... Can you cut and paste one out of a Unix session that worked (the one I did above)? Just to convince me it does work somewhere... By the way, this idea of a "magic cookie" extends into several areas (and carries liabilities, read bugs, with it in many cases). I passed the following along to Roger rectntly, you might enjoy it - I actually should suggest the fix to MS. The puzzle, (actually a test for someone applying for a job claiming Excel expertise). This came to me in the form of a real file from a client that one of our people was pulling hair trying to figure out how to bring into Excel. So -- you talk to the applicant and ask them to demonstrate their knowledge of Excel by doing the following things: 1. Create a new worksheet. (Most applicants should handle this) 2. Enter the following data into A1:B2 (please be exact, at least in entering it) +-----------+-----------+ |IDNUMBER |NAME | +-----------+-----------+ |123 |Lotus | +-----------+-----------+ 3. Save the worksheet as a "tab delimited text" file. 4. Close the worksheet (to demonstrate that you know you have to tell Excel that you really want to...) 5. Reopen (import) the text file to add new data. +-----------+-----------+ |IDNUMBER |NAME | +-----------+-----------+ |123 |Lotus | +-----------+-----------+ |666 |Wizard | +-----------+-----------+ (this is the test) -- Joey From: "Joey K. Tuttle" <jkt@pocus.hexagon.com> To: neitzel@gaertner.de Cc: bugs@gaertner.de, kbi@interlog.com, jhui@interlog.com Subject: Re: j-unix/4: Using the "J shell" jrun as a login shell Date: Sun, 28 Jun 1998 18:00:11 -0700 Hmmm, I expected to hear from you before now... At 19:08 -0700 98/6/22, Joey K. Tuttle wrote: > >I would still like to run a J shell.... Can you cut and paste one out >of a Unix session that worked (the one I did above)? >Just to convince me it does work somewhere... > Does it work anywhere? What environments have actually been tested? -- Joey From: Martin Neitzel <neitzel@gaertner.de> To: jkt@pocus.hexagon.com Cc: bugs@gaertner.de, jhui@interlog.com, kbi@interlog.com Subject: Re: j-unix/4: Using the "J shell" jrun as a login shell Date: Mon, 29 Jun 1998 13:40:25 +0200 (MET DST) Sorry not to have come back to you earlier. > Does it work anywhere? What environments have actually been tested? It does work on Linux. However, I can reproduce that it does not work under SunOS4.x and SunOS5.x. Nor does it under under NetBSD. It appears that those execve(2)s are requiring that the #!-interpreter is itself a true binary, not another #!-script. There are basically two possible solutions: >Fix: (1) re-implement the jrun wrapper as a compiled c program (2) extend jint with a command-line option to work on the script At least (1) should be doable for you as an interim fix. It is not imediatly clear how this should be approached in the next release. By and large, I'd wished there were a few more ways to have J operate on stdin. From: "Joey K. Tuttle" <jkt@pocus.hexagon.com> To: Martin Neitzel <neitzel@gaertner.de> Cc: pocus!jkt@uucp-1.csn.net, bugs@gaertner.de, rhui@interlog.com, kbi@interlog.com Subject: Re: j-unix/4: Using the "J shell" jrun as a login shell Date: Mon, 29 Jun 1998 13:12:49 -0700 At 13:40 +0200 98/6/29, Martin Neitzel wrote: >Sorry not to have come back to you earlier. > >> Does it work anywhere? What environments have actually been tested? > >It does work on Linux. > >However, I can reproduce that it does not work under SunOS4.x and >SunOS5.x. Nor does it under under NetBSD. > >It appears that those execve(2)s are requiring that the #!-interpreter >is itself a true binary, not another #!-script. > Thank you for your insightful reply! I can also add that this stuff doesn't work in - UNIX(r) System V Release 4.2 (as supplied by Novell and I believe now by SCO) I suspect your observation that most Unix systems require that a #!-interpreter be a true binary is correct. I have found no other attributes for things like Perl which do have nice "shell behavior". >There are basically two possible solutions: > >>Fix: > I can try experimenting with these possibilities. I do have a Linux system available, so I can try things there as well. >(1) re-implement the jrun wrapper as a compiled c program >(2) extend jint with a command-line option to work on the script > >At least (1) should be doable for you as an interim fix. It is not >imediatly clear how this should be approached in the next release. >By and large, I'd wished there were a few more ways to have J >operate on stdin. I'm interested in your last statement, I think I agree that it would be good to have good support of Unix stdin - I make a lot of use of what is currently available and it gets me close to what I'm looking for. Please tell me more about what kinds of things you are thinking of. -- Joey From: Martin Neitzel <neitzel@gaertner.de> To: jkt@pocus.hexagon.com Cc: bugs@gaertner.de, kbi@interlog.com, pocus!jkt@uucp-1.csn.net, rhui@interlog.com Subject: Re: j-unix/4: Using the "J shell" jrun as a login shell Date: Tue, 30 Jun 1998 13:24:50 +0200 (MET DST) > I'm interested in your last statement, I think I agree that it would be > good to have good support of Unix stdin - I make a lot of use of what is > currently available and it gets me close to what I'm looking for. Please > tell me more about what kinds of things you are thinking of. J is -- I think -- able to do both but I also think there are rough edges. I don't like the mere handwaving in these sentences, and Roger needs some concrete statements and suggestions in a coherent text. Let's prepare that text and get back to Roger with that. State-Changed-From-To: open-suspended State-Changed-By: neitzel State-Changed-When: Tue Jun 30 15:35:32 MET DST 1998 State-Changed-Why: Suspended until C source version for jrun is available as a temporary fix for jkt. From: Martin Neitzel <neitzel@gaertner.de> To: jkt@pocus.hexagon.com Cc: bugs@gaertner.de, cdburke@interlog.com, eiverson@interlog.com, kbi@interlog.com, rhui@interlog.com Subject: Re: j-unix/4: Using the "J shell" jrun as a login shell Date: Thu, 2 Jul 1998 11:38:40 +0200 (MET DST) Hello Joey, the following simple C program should give you a nice binary "jrun". You have to tune the file paths by hand (the version in the distribution kit takes care of that). ----snip #ifdef __STDC__ #include <stdlib.h> #else #include <malloc.h> #endif main (ac, av) int ac; char **av; { char **nav, **addons; nav = (char**) malloc ((5+ac+1) * sizeof (char*)); nav[0] = "jrun"; nav[1] = "/usr/local/lib/j-401c/system/extras/config/profile.ijs"; nav[2] = "/usr/local/lib/j-401c/system/main/files.ijs"; nav[3] = "/usr/local/lib/j-401c/shell.ijs"; nav[4] = "--"; addons = nav+5; while (*addons++ = *++av) ; if (!getenv("JLIB")) putenv ("JLIB=/usr/local/lib/j-401c"); execv ("/usr/local/bin/jint-401c", nav); } ----snap save as jrun.c and then cc jrun.c -o jrun && strip jrun && mv jrun /usr/local/bin or whatever. FYI, the first dist kits of J4.01c generate a completely botched jrun shell script, corrected kits will be out shortly (read: Friday-Sunday) including the C based jrun. Martin State-Changed-From-To: suspended-feedback State-Changed-By: neitzel State-Changed-When: Thu Jul 2 12:20:42 MET DST 1998 State-Changed-Why: Does the C source fullfill your needs? State-Changed-From-To: feedback-closed State-Changed-By: neitzel State-Changed-When: Fri Jul 3 09:41:03 MET DST 1998 State-Changed-Why: jkt@aptos-1$ uname -a SunOS aptos-1 5.4 generic sun4m sparc jkt@aptos-1$ cat total #! /usr/local/bin/jrun echo +/ > 0".&> }.ARGV jkt@aptos-1$ total 1 2 3 6 jkt@aptos-1$ # YESSSSS!!! >Unformatted: