612 West 115th Street, New York NY 10025 USA • kermit@columbia.edu
| …since 1981 |
| Home | Kermit 95 | C-Kermit | Scripts | Current | New | FAQ | Support |
|
|
The Kermit Project |
Columbia
University
612 West 115th Street, New York NY 10025 USA • kermit@columbia.edu
| ||||||||
| |||||||||
Frank da Cruz
Columbia University
fdc@columbia.edu
Current test level: 9.0.299 Alpha.03
Date: 19 March 2010
This page updated: Fri Mar 19 17:04:09 2010 EDT
Work on C-Kermit has continued, on and off, since the release of C-Kermit 8.0.211 on 10 April 2004. The working version was called 8.0.212 but it will be released as C-Kermit 9.0, starting with some Alpha-test releases, then some Betas, then the final release when it's ready. The major goals of this release are (a) stability; (b) compatibility with newer OS releases and hardware; (c) support for large files and 64-bit integers on as many platforms as possible; and (d) a great many VMS-specific improvements. Other new features are listed below. CLICK HERE to access and download the latest build.
Thanks to (at least) Jeff Altman, Nelson Beebe, Gerry Belanger, Rob Brown, Christian Corti, John Dunlap, Peter Eichhorn, Carl Friedberg, Günter Knauf, Jason Lehr, Mike Rechtman, Steven Schweda (SMS), Kinjal Shah, Andy Tanenbaum, Seth Theriault, Martin Vorländer, and Eric Weaver for assistance, and to Hewlett-Packard Company for support.
- What's New in Alpha.03
- Demonstration: HP Switch Configuration Backup
- Demonstration: HP iLO Blade Configuration
- Demonstration: IBM/Rolm/Siemens CBX Management
- Platforms
- Large Files
- How to Test Large-File Transfer
- Arithmetic with Large Integers
- Other New Features
- Incompatibilities
- What's Not In C-Kermit 9.0
- And a Loose End
$ @ckvker "" "" "CK_SSL"the OpenSSL version is detected automatically and the appropriate compile-time options are emitted (such as OPENSSL_DISABLE_OLD_DES_SUPPORT).
Special thanks to Steven M Schweda, Martin Vorländer, and Rob Brown for most of the VMS improvements in Alpha.03.
Typically a backup can be done by making a Telnet, SSH, or serial connection to the device with Kermit and giving a command such as "show config" at the command-line prompt of the device with Kermit's session log activated. The result is a list of the commands that were used to establish the current configuration, suitable for feeding back to the device's console to reestablish the same configuration or to duplicate it on another device.
At an HP installation it was noted, however, that while the HP switches (various ProCurve models) produced the desired commands, they were interspersed with escape sequences for special effects, thus rendering the recorded sessions unsuitable for feeding back into the switches.
C-Kermit 9.0 introduces a new feature to strip the offending sequences out of a session log, leaving just the text. The command SET SESSION-LOG TEXT activates this feature. In C-Kermit 9.0 Alpha.02 and earlier, escape sequence stripping occurred only while logging interactive (CONNECT) sessions; beginning with Alpha.03 it is done also for data that is read by INPUT commands.
A sample script is HERE, and its data file is HERE. This script also illustrates some other new features of Alpha.03:
$ DEBUG=1 scriptname arg1 arg2...and then include the following command in your script:
if defined \$(DEBUG) set debug message on
which, if the SET HOST command fails, prints "FATAL - set host somehost.somecompany.com" and then exits with status 1 (which normally indicates failure).set host somehost.somecompany.com if fail exit 1 "FATAL - \v(lastcommand)"
| * | Of course the University is deploying new technology but the but the old system will be used in parallel for some time to come. |
The biggest change since C-Kermit 8.0.211 is support for large files on platforms that support them. A "large file" is one whose size is greater than 231-1 (2,147,483,647) bytes (2GB-1); that is, one whose size requires more than 31 bits to represent. Before now, Kermit was able to access such files only on 100% 64-bit platforms such as Digital Unix, later known as Tru64 Unix. In the new release, Kermit takes advantage of the X/Open Single UNIX Specification Version 2 (UNIX 98) Large File Support (LFS) specification, which allows 32-bit platforms to create, access, and manage files larger than 2GB.
Accommodating large files required code changes in many modules, affecting not only file transfer, but also file management functions from directory listings to local file manipulation, plus the user interface itself to allow entry and display of large numbers. All this had to be done in a way that would not affect pure 32-bit builds on platforms that do not support large files. Large file support is summarized in the Table of Platforms; entries in Yellow (32-bit builds that support 64-bit integers) and Green (64-bit builds) support large files.
Note that VMS C-Kermit and Kermit 95 for Windows have always been able to transfer large files. However their user interface used 32-bit integers for statistics and the file transfer display. In C-Kermit 9.0 Alpha.03, VMS C-Kermit on 64-bit platforms (Alpha and Itanium) should now give correct statistics and progress displays. (We'll see about Kermit 95 later.)
(At remote kermit...)
$ kermit -Y
C-Kermit> receive /calibrate
(Return to local kermit...)
Ctrl-\c
C-Kermit> send /calibrate:4300000000
This sends a simulated file 4.3GB in length, that does not exist on the sender and will not take up any disk space on the receiver. SEND /CALIBRATE: accepts big numbers only in Kermit versions that support them (this does not include Kermit 95 on Windows).
S-Expressions can now be forced to operate with integers only, without floating-point conversion or having to explicitly truncate each result; as an example. see the revised Easter date calculation script.
ioption in P1 (and by including
CK_SSLin P3 if you also want SSL, and then also
OPENSSL_DISABLE_OLD_DES_SUPPORTif necessary). Much testing is needed to determine if it should be included in the final C-Kermit 9.0 release.
foption in P1.
dir /top:10 /sort:size /reverse *or equivalently, "hdir /top:10 *". WDIR lists files in reverse chronological order, shorthand for "dir /sort:date /reverse".
quantity description...in which the first "word" is a number, and a description (for example, the name of an item). Here is how to use FSEEK to quickly get the total quantity of any given item, which is passed as a parameter (either a literal string or a pattern) on the command line:
#!/usr/local/bin/kermit +
if not def \%1 exit 1 Usage: \fbasename(\%0) string-or-pattern
.filename = /usr/local/data/items.log # Substitute the actual filename
set case off # Searches are case-independent
fopen /read \%c \m(filename) # Open the file
if fail exit 1 "\m(filename): \v(errstring)" # Fail: exit with error message
.total = 0 # OK: Initialize the total
echo Searching "\%1"...
while true {
fseek /line /relative /find:\%1 \%c 0 # Get next line that has target
if fail break # Failure indicates EOF
fread /line \%c line # Read it
if fail break # (shouldn't happen)
increment total \fword(\m(line),1) # Increment the total
}
fclose \%c # Close the file
echo Total for "\%1" : \m(total) # Print the result
exit 0
The syntax of the FSEEK command in this example indicates that each search
should start relative to the current file line. Since Kermit is an
interpretive language, FSEEK is a lot faster than FREAD'ing each line and
checking it for the target, especially for big files. An especially handy
use for FSEEK is for use with potentially huge sequentially timestamped
logs, to seek directly to the date-time where you want to start processing.
Some other improvements for the FOPEN/FREAD/FWRITE/FCLOSE family of commands
are included also (performance, bug fixes, convenience features), listed in
the change log. (Prior to 9.0.299 Alpha.02,
the FSEEK /FIND: command always started from the top.)
There's one exception this time. The \fsplit() function is incredibly handy, it can do almost anything, up to and including parsing a LISP program (the underlying code is the basis of the S-Expression interpreter). But did you ever try to use it to parse (say) a Tab-Separated-List (TSV file) or Comma-Separated-List (CSV)? It works as expected as long as the data contains only 7-bit characters. But if your data contains (say) Spanish or German or Russian text written in an 8-bit character set such as ISO 8859-1, every 8-bit character (any value 128-255) is treated as a break character. This is fixed in C-Kermit 9.0 by treating all 8-bit bytes as "include" characters rather than break characters, a total reversal of past behavior. I don't think it will affect anyone though, because if this had happened to anyone, I would have heard about it!
Since most standard 8-bit character sets have control characters in positions 128-160, it might have made sense to keep 128-160 in the break set, but with the proliferation of Microsoft Windows code pages, there is no telling which 8-bit character is likely to be some kind of text, e.g. “smart quotes” or East European or Turkish accented letters.
That's because as coded (through 8.0.211), C-Kermit simply starts the external protocol in a fork with its standard i/o redirected to the connection. This completely bypasses the encryption and decryption that is done by C-Kermit itself, and of course it doesn't work. The same thing occurs if you use the REDIRECT command. The routine that handles this is ttruncmd() in ckutio.c.
In order to allow (say) Zmodem transfers on secure connections, it is necessary for C-Kermit to interpose itself between the external Zmodem program and the connection, decrypting the incoming stream before feeding it to Zmodem and encrypting Zmodem's output before sending out the connection.
In principal, this is simple enough. We open a pseudoterminal pair ("master" and "slave") for Zmodem's i/o and we create a fork and start Zmodem in it; we read from the fork pty's standard output, encrypt, and send to the net; we read from the net, decrypt, and write to the fork pty's standard input.
In practice, it's not so simple. First of all, pseudoterminals (ptys) don't seem to interface correctly with certain crucial APIs, at least not in the OS's I have tried (Mac OS X, Linux, NetBSD, etc), such as select(). And i/o with the pty often – perhaps always – fails to indicate errors when they occur; for example, when the fork has exited.
But, even after coding around the apparent uselessness of select() for multiplexing pty and net, and using various tricks to detect when the external protocol exits and what its exit status is, I'm still left with a show-stopping problem: I just simply can not download (receive) a file with Zmodem, which is the main thing that people would probably want to do. I can send files just fine, but not receive. The incoming stream is delivered to Zmodem (to the pty slave) but upon arrival at the Zmodem process itself, pieces are always missing and/or corrupt. Yet I can receive files just fine if I use Kermit itself (C-Kermit or G-Kermit) as the external protocol, rather than Zmodem.
I can think of two reasons why this might be the case:
But Zmodem puts its controlling terminal into raw mode. And C-Kermit puts the pty into raw mode too, just for good measure. If any 0xFF codes are in the Zmodem data stream, and it's a Telnet session, Kermit does any needed byte stuffing/unstuffing automatically. Anyway, if I tell Zmodem to prefix everything, it makes no difference.
I can vary the size of the i/o buffers used for writing to the pty, and get different effects, but I am not able to get a clean download, no matter what buffer size I use. write()'ing to the pty does not return an error, and I can't see the errors because they happen on the master side. It's as if the path between the pty slave and master lacks flow control; I deliver a valid data stream to the pty slave and the master gets bits and pieces. This impression is bolstered somewhat by the "man 7 pty" page in HP-UX, which talks about some special modes for ptys that turn off all termio processing and guarantee a flow-controlled reliable stream of bytes in both directions – a feature that seems to be specific to HP-UX, and exactly the one we need everywhere.
Well, in Pass One I used C-Kermit's existing pty routines from ckupty.[ch], which are well-proven in terms of portability and of actually working. They are currently used by SET HOST /PTY for making terminal connections to external processes. But these routines are written on the assumption that the pty is to be accessed interactively, and maybe they are setting the fork/pty arrangement up in such a way that that's not suitable for file transfer. The Pass One routine is called xttptycmd() in ckutio.c.
So in Pass Two I made a second copy of the routine, yttptycmd(), that manages the pty and fork itself, so all the code is in one place and it's simple and understandable. But it still doesn't work for Zmodem downloads. In this routine, I use openpty() to get the pty pair, which is not portable, so I can have access to both the master and slave pty file descriptors. This version can be used only a platforms that have openpty(): Linux, Mac OS X, NetBSD, etc.
In Pass Three, zttptycmd(), I tried using pipes instead of ptys, in case ptys are simply not up to this task (but that can't be true because if I make a Telnet or SSH connection into a host, I can send files to it with Zmodem, and the remote Zmodem receiver is, indeed, running on a pty). But pipes didn't work either.
In Pass Four, I extracted the relevant routines into a standalone program based on yttptycmd() (the openpty() version, for simplicity), which I tested on Mac OS X, the idea being to rule out any "environmental" effects of running inside the C-Kermit process. There was no difference -- Kermit transfers (with C-Kermit itself as the external protocol) worked; Zmodem transfers (neither sz or lsz) did not.
Well, it's a much longer story. As the external protocol, I've tried rzsz, crzsz, and lrzsz. We know that some of these have quirks regarding standard i/o, etc, which is one of the reasons for using ptys in the first place, and i/o does work – just not reliably. Anyway, the 1100 lines or so of ckc299.txt, starting just below where it says "--- Dev.27 ---" tell the full story. At this point I have to give up and move on; it might be more productive to let somebody else who has more experience with ptys take a look at it – if indeed anyone still cares about being able to do Zmodem transfers over secure Telnet connections.
C-Kermit 9.0 contains the three new routines (and some auxiliary ones), but they are not compiled or called unless you build it specially:
make targetname KFLAGS=-DXTTPTYCMD (builds with xttptycmd())
make targetname KFLAGS=-DYTTPTYCMD (builds with yttptycmd())
make targetname KFLAGS=-DZTTPTYCMD (builds with zttptycmd())
These are all in ckutio.c. As noted, the second one works only for Linux, FreeBSD, NetBSD, and Mac OS X, because it uses non-POSIX, non-portable openpty(). If you want to try it on some other platform that has openpty(), you can build it like this:
make targetname "KFLAGS=-DYTTPTYCMD -DHAVE_OPENPTY"
(and let me know, so I can have HAVE_OPENPTY predefined for that platform too). The best strategy to get this working, I think, would be to concentrate on yttptycmd(), which is the simpler of the two pty-based routines. If it can be made to work, then we'll see if we can retrofit it to use the ckupty.c routines so it will be portable to non-BSD platforms.
By the way, if you build with any of [XYZ]TTPTYCMD defined, then the selected routine will always be used in place of ttruncmd(). This is to allow testing on all kinds of connections, not just secure ones, in both local and remote mode. Once the thing works, if it ever does, I'll add the appropriate tests and/or commands.
By default, in the initial test release, C-Kermit 9.0 uses ttruncmd() on serial connections and ttyptycmd() on network connections. Even when a network connection is not encrypted, Kermit still needs to handle the network protocol, e.g. the quoting of 0xff bytes on Telnet connections.
– Frank da Cruz fdc@columbia.edu
| Home | Kermit 95 | C-Kermit | Scripts | Current | New | FAQ | Support |