The 2!:2 HostIO foreign

The 2!:2 "Host IO" foreign verb is used to run some Unix process in parallel with the J interpreter, communicating with it through two file descriptors. (I.e., two pipes.)

It is only minimally documented in the DoJ, and it is easy to use it incorrectly if you are not versed in the unix way of life. So here is a full blown example of its proper use.

NB. We are running: 9!:14 '' 4.05/2000-11-13/11:30 NB. Fork a tr(1) process to translate characters from lower to upper case. NB. The three integers returned are: NB. NB. - the process id NB. - a J file number representing the process' standard output NB. - a J file number representing the process' standard input NB. NB. Unfortunately, the DoJ lists the file numbers in arguably wrong order. NB. A cross check with the (correctly documented) "<>" markers NB. in the 1!:20'' output establish their correct use (as demonstrated NB. here). [ 'pid from to' =. 2!:2 'tr a-z A-Z' 27963 136494568 136494656 NB. Let's check whether our concurrent "tr" process is really running NB. and that the pid is correct: 2!:0 'ps' PID TTY STAT TIME COMMAND 26495 a2 S 0:00 -bash 27444 a2 S 0:00 jconsole 27963 a2 S 0:00 tr a-z A-Z 27965 a2 S 0:00 sh -c ps > /tmp/27444eaa 27966 a2 R 0:00 ps pid 27963 NB. The 1!:20 foreign returns all J file numbers currently in use, NB. including the numbers referring to processes instead of files. NB. Processes are not listed with their pid but the actual command, NB. including a small </> arrow marker indicating the direction NB. the data flows on this channel. (Ain't that nice?!) 1!:20'' +---------+-----------+ |136494568|<tr a-z A-Z| NB. output of (=coming from) the command +---------+-----------+ |136494656|>tr a-z A-Z| NB. input for (=going to) the command +---------+-----------+ to 136494656 NB. Hey, we got it right! NB. Send a few chacters to the process, i.e., write to the NB. file number referring to the process' input. NB. Because this is just a simple example, this is all NB. we want to write and we close the write channel right away. NB. Pipes between unix processes are usually buffered (historically NB. at 4 KB), and closing the channel makes all data available NB. to the "tr" process (which is otherwise not eager to get its NB. input data any earlier): 'hello ' 1!:2 to NB. write 'world!' 1!:2 to NB. write 1!:22 to NB. close 1 1!:20'' NB. check after close of the write channel: +---------+-----------+ |136494568|<tr a-z A-Z| +---------+-----------+ NB. At this moment, a few things happened behind the scenes: NB. NB. - with the close, the buffered 'hello world!' data was flushed NB. to the "tr" process. NB. - the "tr" process submitted its "HELLO WORLD!" to its stdout, NB. i.e. it is now buffered in the pipe connected to the J process. NB. - the "tr" got noticed of the EOF of its stdin; NB. so it has it's work done, closes it's stdout, NB. end exit(2)s with return value 0 ("success"). NB. We can now read the data: 1!:1 from HELLO WORLD! NB. Such a 1!:1 read reads and returns "everything until the EOF". NB. Further reads just return an empty string: 1!:1 from NB. zero characters here NB. (= NEWBIE BEWARE!): NB. If we would have launched the "1!:1 from" read directly after NB. the 'hello ' or 'world!' writes, it would actually hang in NB. a "blocked read", waiting seemlingly forever: NB. NB. - the 1!:1 waits for the finished "tr" output, NB. - the "tr" output won't be finished until the "tr" *input* NB. is finished, NB. - and it this point we cannot finish that input (with "1!:22 to", NB. remember?), because we already hang in the 1!:1 read - D'oh! NB. (Ctrl-C to the rescue if that happened to you.) NB. We just happen to know that we have received all data. NB. So we close the channel for reading, too, and check that NB. no open files are listed anymore: 1!:22 from 1 1!:20'' NB. returns shape 0 2 empty table NB. We are almost done now. NB. Some tiny detail has still to be taken care of, though: NB. NB. A unix process which has exited (our "tr") is still managed by NB. the unix system until its parent process asks about its exit code, NB. using the wait(2) system call. Until then, the (already exited) NB. child is listed as a "zombie" process (state Z): 2!:0 'ps' PID TTY STAT TIME COMMAND 26495 a2 S 0:00 -bash 27444 a2 S 0:00 jconsole 27963 a2 Z 0:00 (tr <zombie>) 28230 a2 S 0:00 sh -c ps > /tmp/27444faa 28231 a2 R 0:00 ps NB. So let's "wait" for the end/exit of our process. NB. Since is has exited already, this wait will instantly NB. return: 2!:3 pid 0 NB. The unix process list is now "clean": 2!:0 'ps' PID TTY STAT TIME COMMAND 26495 a2 S 0:00 -bash 27444 a2 S 0:00 jconsole 28304 a2 S 0:00 sh -c ps > /tmp/27444iaa 28305 a2 R 0:00 ps

This completes our simple example.

Expect buffering to become an issue when you intend an interlocked cooperation between J and the other process.

You can only control the behaviour on the J side of the pipes:

• Writing small chunks is easily enough done with 1!:2 (we did that already in the example above and issued that 'hello world!' in two parts.
• You can use the 1!:11 "indexed read" foreign to read data from the process in smaller chunks instead of "everything until the EOF".

Still, if you intend to...

1. write 'hello' to the "tr" process,
2. read 'HELLO' back from the process,
3. write 'world' to "tr",
4. read 'WORLD' back from "tr",

you'll notice that tr(1) doesn't give a damn to read its input in such little chunks. It's simply not written that way. There is nothing the J system or your J script could do about that.

Most unix tools read input in terms of lines when stdin is a terminal, but in larger blocks when stdin is file or pipe (our case). Likewise, output can more or less buffered, depending on the destination. Then again, Unix programs already designed to "speak" some protocol should be easy to deal with.

The possible buffering between processes is good thing. It allows a "producer" process writing into a pipe to plod along even when the receiving process is a bit slow to pick up the data, or was simply not yet scheduled to run. Only when the recipient process is so slow that the 4K buffer gets full, the sender will have to slow down, too. It will be blocked when attempting its next write to the pipe, until the buffer has been sufficiently emptied again (by the receiving precoess. This is a healthly compromise.

If you don't do the 2!:3 wait thing, the world won't stop turning. When the jconsole process (as parent of the zombie process) exits, the zombie becomes shortly an orphan and quickly "reparented" to the unix "init" process. This does proper wait(2)s on all its processes, so the zombie gets cleanly discarded when J exits. Zombies in a system's process list are seen as an indication for sloppy programming, though, and a bit frowned upon.