****************************************************************************** * * * * * Eichstaett-to-Beta-Code-Text-Conversion-Program * * * * (c) B.Meissner, 1992-2010 * * * ****************************************************************************** Preface: New Version ==================== mc has been ported to C-SNOBOL (Phil Budne), because of this system's wide acceptance. mc has been augmented to include all the extended codes which have been introduced into the Eichstaett text files, long line conventions and other oddities. All I/O routines in mc. have been changed from SPITBOL conventions to SNOBOL4. It would be easy to revert this at some stage (or introduce system-aware alternatives as we did in betautf8.sno, which is completely transparent to all variants of the SNOBOL4 language), but given the rarity with which mc is probably run, we did not want to invest time on this. Thus you should use snobol4 -b -d 4096k -S 500k -P 500k ./mc.sno instead of mc.exe. Snobol4 can be retrieved in source format from: http://www.snobol4.org/csnobol4/curr We provide a couple of SNOBOL4 versions in the snobol4.dir directory. There are calling examples in snobol4.bat, mc.bat and mc. MC and Errors in the Text Files =============================== mc cannot, however, correct errors which arre present in the Eichstaett text files themselves. In the most recent versions (2008 and 2009/2010), there are three such errors which should be corrected at some stage: 1) ILS_III2 contains erroneous lines and line lengths in text no. 9259a. These errors make the text completely incomprehensible, even to the ConcEyst program itself. The ConcEyst program's rendering of the text is documented in the screenshot "fehler.png" and shows that something must have gone very wrong with the codes. In beta code, this text looks like: 98.9259a.184.108.40.206.1 $:E)\A(|U)\& undoc. `0 $*SHA)=*|T& undoc. `0 $*R&r[1centuria]1$I)HU)\;[1centuria]1$*T& undoc. `0 $*E&r$E)\A(| H&[1milliaria]1$*T:&[1milliaria]1$*A*E&g$*SI*)T& undoc. `0 $*S*T&[1milliaria]1O$*T:&[1milliaria]1$*AHA*(| T&[1milliaria]1x$*E&[1milliaria]1$U)\E)\A*(|R&dSVO AT EXORNATIONEM$A)=| &BALINEI$H\|&DONO DEDIT Note that mc inserts an " undoc. " tag, followed by the numerical value of the code in error (in this case: zero=0) to make identification of such errors easier. 2) SEG_52 seems as if there was an error after inscription number 1021, since ConcEyst does not display any inscriptions after that. The reason for this seems to a weird ordering of the texts in the .NDX file for this file: The texts are not ordered according to their numbers, but to their respective dates. This inconsistency is corrected on-the-fly by mc, which sorts the texts according to their numbers. Mc uses a complicated queue-structure, a multiply linked list to re-arrange the texts. 3) There are a couple of files which are referred to differently internally and externally by ConcEyst. These are the McCrum, Meiggs&Lewis and Rhodes_Osborne files which have differing uppercase/lowercase renderings of their file names in the corresponding table files. On case-aware operating systems (all modern systems are!) this may lead to errors. A small unix shell script provides a remedy by externally linking the files to symbolic names which more precisely correspond to the names these files have in the internal data bases ("rename_files"). Thus, mc cannot be used only to make the ConcEyst text files accessible by beta-code-aware programs (like V&F), but also to check and debug the ConcEyst text files themselves. Purpose ======= The Catholic University of Eichstaett, Germany (Prof. Dr. J.Malitz) provides a set of Greek and Latin epigraphical corpora on disk. They come with concordance routines and browser facilities, but the files use a special format. To adapt them to the V&F program for beta-coded texts to get more liberty in using the texts, a conversion routine is therefore necessary. For this purpose, Burkhard Meissner (University of Halle) has written the MC conversion program that performs the task. MC converts the Eichstaett files in beta code format to be read by Burkhard Meissner's V&F program. Other programs which use beta-coded text files (as distributed by the Thesaurus Linguae Graecae and the Packard Humanities Institute) might also be used to view and search the resulting text files, but the program has been specifically designed for the V&F environment. Since there is no printed documentation for MC.EXE, you should print out this file on your standard line printer to get a reading copy: COPY MC.DOC LPT1 Legal Conditions ================ MC.EXE forms an integral part of V&F. All legal conditions that apply to the other software items of the V&F package apply to MC.EXE, too: 1) Conditions of Use MC is a program that its author, Burkhard Meissner, has written privately for his own purposes. However, the author of the MC software grants the right to use that software to any legitimate user of MC, under the following terms and conditions: A legitimate user is defined as any natural or legal person that is already a legitimate user of the V&F software. This entails that the user may have paid the license fee of 1000 DM (institutions) or 800 DM (individuals) to Burkhard Meissner for the use of the V&F software or ob- tained the software by downloading from the Ancient History home page at the the Helmut-Schmidt-University web site. The legitimate user does not obtain the software as his property; instead, all rights of copying and distributing the software remain with the author. However, the legitimate user acquires the right to use the software on any single machine at a time and to make as many backup copies for his own purposes as he thinks appropriate. The legimtimate user being a non-private institutional (i.e.: legal) person, it may use the software on any machine that is physically contained within the institution's building. Therefore, the right to use the MC software is transferred under the condition that the aforementioned fee is paid to the author, and the right to use the MC software is transferred only to the person that has paid the fee. Using MC in an networking environment or on different machines within one physically coherent institution does conform to these conditions of legitimate use of MC; copying and distributing MC to more than one separate machine to use the software on more than one machine by different persons at the same time does not. The author of the MC software, Burkhard Meissner holds and will hold the copyright of the software, and he is its material, intellectual and spiritual owner. This entails that users have to give due credit to the author and his MC software when they publish results produced from the use of MC. Therefore, the right to use MC is transferred under the explicit condition that the user mentions the MC software and its author in any publication that results from serious use of the MC software. Any use of the MC software that is not rendered legitimate according to these conditions is considered illegitimate and illegal. 2) Warranty Conditions a) Limited Warranty The MC software is provided "as is" without warranty of any kind, either expressed or implied, including, but not limited to the implied warranties of merchantability and fitness for a particular purpose. The entire risk as to the quality and performance of the program is with the user. Should the program prove defective, the user assumes the entire cost of all necessary servicing, repair or correction. The author, Burkhard Meissner, does not warrant that the functions contained in the program will meet the user's requirements or that the operation of the program will be uninterrupted or error-free. However, he warrants the diskette(s) on which the program is furnished, if it is so furnished, to be free from defects in materials and workmanship under normal use for a period of ninety (90) days from the date of delivery to the user. b) Limitations of Remedies The author's entire liability and the users exclusive remedy shall be: 1) The replacement of any diskette not meeting our "Limited Warranty" and which is returned to the author together with a copy of the user's receipt, or 2) if the author, Burkhard Meissner, is unable to deliver a replacement diskette which is free of defects in materials or workmanship, the user may terminate the Agreement between author and user by returning the software, upon which his money shall be refunded. In no event will the author, Burkhard Meissner, be liable to the user for any damages, including any lost profits, lost savings or other incidental or consequential damages arising out of the use or inability to use the program even if the author has been advised of the possibility of such damages, or for any claim by any other party. Using MC ======== To use this program, you should know something about the organization of the Eichstaett text files: Each text corpus is organized around a corpus (.CPS) file which contains pointers to a set of index (.NDX) and text (.TXT) files. In order to collect all texts of such a corpus into one beta code file, one has to resolve the file pointers in the corpus file and the pointers to the single texts in the .NDX files. Our program does all this for single corpora. This means: You must re-run it for each .CPS file to obtain a full conversion of all existing text files. It assumes that the .TXT, .NDX and .CPS which belong together are located in the present directory and writes its output into the same directory. It creates AUTHTAB.DIR automatically, the beta code author list. Suppose, you have DFG.CPS with its adjacent files. To create a beta code file set CV_001, start our conversion program MC: MC Then enter the name of the input file (e.d.: DFG) and the output file (e.d.: CV_001). For Latin texts, enter Y upon the third question. The Y overrides the default Greek font of the Eichstaett files. This is necessary with texts which are mainly in Latin. As a new feature, you can then enter a string of up to six reference level names which will be used instead of the default strings (Inscription.....line). E.d.: Inscription....section line Confirm with Y, or else re-enter all informations. The program will create CV_001.txt, CV_001.idt and AUTHTAB.dir. Be sure there are no files with the same names which you do not want to be overwritten. An ongoing MC conversion process can always be interrupted using the Ctrl-Break key. In this case you should allow some time to the MC.EXE virtual memory manager system to restore the memory state to DOS standards, close the swap file and close the files written so far. When the process finishes, you can start V&F, make the directory, where the new files are, your CDdrive directory (Options menu, number 6, do not forget the back slash), Re-Read the author list and work on this file. To append the new files to existing sets of beta code texts, you may use our AUTHCONV utility with the /A switch and the DOS APPEND command (see below). There are no restrictions as to the use of the resulting files; they can be read, searched and indexed like any other TLG or PHI beta code text. The program sorts the texts by names. It uses a special algorithm which compares the text names with respect to the positional value of their first numeric component, without comparing the text names only alphabetically. Thus it is ensured that 400a sorts after 400 but before 4000. A text name like 2347-9 sorts after 2347 but before 2349. The complicated string and list conversion operations during the sorting process make it quite slow with large text collections. In addition, the memory requirements are gigantic. There is no way of running it on a 1 MByte machine; a machine with only 2 or 3 MBytes may come into serious difficulties with excessive swapping. For practical applications, 4 MBytes seem the minimum. MC - An Example Session ======================= Suppose you have a set of Eichstaett files. These come as text files (.TXT), index files (.NDX) and corpus files (.CPS). The dictionary files (.DCT and alike) are of no interest to the conversion process and are not needed for MC (but you must not delete them: They are necessary for the Eichstaett concordance routines themselves). You should copy all these files in one sub directory (for security reasons, you should perform the conversions, using copies of the files, and not the original ones). Now you should detemine how many and which .CPS files there are: DIR *.CPS For security reasons, create a sub directory for each single .CPS file with its distinct name (but empty as yet). Suppose you have .CPS files ann_epig.cps 73 14.12.91 11:09 dfg .cps 109 10.09.92 16:54 germania.cps 118 22.01.88 17:04 ils .cps 82 23.04.91 14:56 Now create sub directories: MD ann_epig MD dfg MD germania MD ils Now start the conversion process. Call MC from the DOS prompt and enter "ann_epig" when being asked for a .CPS file name. Enter a distinct name for the output files when asked (e.d.: "cv_001"). You should know, if most of the texts are in Greek or in Latin. If they are in Greek, you should simply press <ENTER> when being asked, if the texts should be forced to have Latin default font. If most of them are in Latin, enter "Y" or "y" as the answer to this third question. This makes the program enter the "l" language code in the resulting files instead of "g". This, in turn, allows for much less language codes within the texts themselves, with the consequence of making them more compact and all V&F processes upon them faster. To start the process, enter "Y" or "y" upon the fourth question, any other letter if you want to make corrections. In this case, you must start MC anew. In the above example, MC converts the texts and writes three files: CV_001.TXT CV_001.IDT AUTHTAB.DIR For security reasons, you should copy (or move, if you have a directory entry mover like UNIX' mv) these files to your ANN_EPIG sub directory: COPY CV_001.TXT ANN_EPIG COPY CV_001.IDT ANN_EPIG COPY AUTHTAB.DIR ANN_EPIG The same processes should be performed for all four .CPS files separately. Of course, you should select divers file names for the output files (e.d.: CV_002 CV_003 CV_004 and so forth). After the last conversion has been performed (CV_004 being the last output in our above example), you should have all resulting files in the sub directories (four in our case). Now copy the files of the first sub directory in your present directory: COPY ANN_EPIG\*.* Now you have AUTHTAB.DIR (with one entry) in your present directory. Call AUTHCONV with this file specified AS THE OUTPUT FILE and the single AUTHTAB.DIR files within the other sub directories as INPUT FILES: AUTHCONV c:\DFG\AUTHTAB.DIR AUTHTAB.DIR /A AUTHCONV c:\GERMANIA\AUTHTAB.DIR AUTHTAB.DIR /A AUTHCONV c:\ILS\AUTHTAB.DIR AUTHTAB.DIR /A Finally, you should move or copy all .TXT and .IDT files as well as the AUTHTAB.DIR file to one separate sub directory where they can be accessed by V&F: MD D:\MALITZ XCOPY ANN_EPIG\*.IDT D:\MALITZ XCOPY ANN_EPIG\*.TXT D:\MALITZ XCOPY DFG\*.IDT D:\MALITZ XCOPY DFG\*.TXT D:\MALITZ XCOPY GERMANIA\*.IDT D:\MALITZ XCOPY GERMANIA\*.TXT D:\MALITZ XCOPY ILS\*.IDT D:\MALITZ XCOPY ILS\*.TXT D:\MALITZ COPY AUTHTAB.DIR D:\MALITZ The old files in the four sub directories can be deleted: DEL ANN_EPIG DEL ILS DEL DFG DEL GERMANIA DEL CV_*.* DEL AUTHTAB.DIR If you want to have these files available together with one of your CD-ROM texts, you should use the DOS command APPEND. Insert your CD into the CD drive. Let's suppose, the drive be G:. Change to the drive, where your new inscription files are located: CD C:\MALITZ Now create a common AUTHTAB.DIR for both sets of texts. Keep the old ones for reference: AUTHCONV G:\AUTHTAB.DIR AUTHTAB.DIR /A Now use the DOS command APPEND to make both directories accessible: APPEND G:\ You can append multiple directories, separated by ";". In your V&F profile PROFILE.V&F you should make 'C:\MALITZ\' your CDdrive text file directory. In this case you will have both sets of texts available at one time. If you insert the DOS APPEND command in your V&F.BAT file to start the program, you can automate the process for the future. To revoke the APPEND assignments, give the command with only one ";": APPEND ; This should be given in your V&F.BAT batch file before returning you to the DOS prompt. Using this strategy, we have connected all TLG and PHI texts with the Malitz texts at Erlangen in one large text file inventory. MC - Technical Details and Possible Problems ============================================ MC uses complicated lists with a multiple-pointer structure to store work lists, text lists and single texts in memory. To sort the texts, these structures must temporarily be converted to linear arrays to be sorted and re-created afterwards. This uses large amounts of memory and takes much time. Once the text lists are sorted and the conversion process is under way (i.e. when the CONVERTING ... lines flicker on the screen), your computer's memory is enough for the process. If, however, you get the SYSTEM ERROR #204 message, informing you of the presence of insufficient memory, your present configuration cannot be used for the job. In this case, you should use MODXCONF as described in the V&F documentation. Just apply it to MC (MODXCONF MC.EXE) and follow the menu instructions. Increase the region size and/or virtual memory size. The swapping process of the virtual memory manager uses large amounts of time and hard disk space. Therefore we have decided to set the MC.EXE region size to only 3 MBytes. This should be enough for most processes. As yet we have found that all conversion processes could be fitted in a memory region that was larger than 1456 KBytes and smaller than 1584 Kbytes. With MC.EXE, you have twice that amount of free space which should be enough by all standards. Acknowledgements ================ The development of MC has been made possible by Prof. Dr. Jaergen Malitz, Eichstaett Catholic University, who disclosed his file formats, provided ample documentation upon the internal structure of his texts and of the programs to handle them. He also invested much time in beta-testing MC. Some errors have been found during this activity, and the program owes many features to Professor Malitz' suggestions. Professor Malitz' work on the computerization of Greek inscriptions of Asia is to be commended for its yielding positive results. Mark Emmer of Catspaw, Inc. has been exceptionally cooperative in responding to demands for improving his already excellent programming tools (SPITBOL-386, SPITBOL-8088). MC Application Error Messages and Numbers ========================================= 1 INPUT .CPS FILE CANNOT BE OPENED 2 OUTPUT FILE CANNOT BE OPENED 3 INPUT .CPS FILE DOES NOT HAVE CORRECT FORMAT 4 INPUT .CPS FILE CORRUPT: NON-MATCHING NUMBER OF ENTRIES 5 INPUT .NDX FILE CANNOT BE OPENED 6 INPUT .NDX FILE CORRUPT: ERRONEOUS FILE FORMAT 7 INPUT .NDX FILE CORRUPT: NON-MATCHING NUMBER OF ENTRIES 8 INPUT .TXT FILE CANNOT BE OPENED 9 INPUT .TXT FILE CORRUPT: TEXT OFFSET CANNOT BE REACHED 10 INPUT .TXT FILE CORRUPT: MISSING END MARKER (FFh) 11 INPUT .TXT FILE CORRUPT: ERRONEOUS LINE LENGTH 12 OUTPUT .TXT FILE CANNOT BE WRITTEN TO 13 OUTPUT .NDX FILE CANNOT BE WRITTEN TO 14 AUTHTAB.DIR CANNOT BE OPENED FOR WRITING 15 CONFLICTING I/O FILE NAMES Technical Details ================= MC used to be built with the INTEL DOS extender (DPMI-compatible). This meant, it could be run on 32-bit DOS machines only, and not under DESQView. Since MC uses large amounts of memory, this seemed reasonably. Nowadays, machines are much faster and much larger: An interpreter implementation is feasible under these circumstances.