--DRAFT DRAFT DRAFT-- --rg Why we need a new cold loader, how we are going to get it, and what difference it will make. why we need a new one. The current FLEABIT FALCON cold loader has a number of gross structual shortcommings. For example, it makes no list structure at all! As a result, things have to go thru two very fragile stages before the package system can even be installed. In the first stage [COLD], no symbols, etc are available at all! This causes simple looking code to fail in strange ways (clobberring NIL), etc. In the next stage [WARM] there is a completely simpleminded INTERN which involves a single OBLIST, absolutely no package structure, etc. Interning things in this way can lead to all kinds of screws later (if package inheritance fails to happen, etc etc). Finally, when the package system comes up, things have to be hacked grossly to be consistant with the multiple extraneous packages, etc which have been introduced in the FLEABIT and FLEABIT-COMPILED systems on the lambda. This obviously needs to be flushed. There are a number of other gross deficiencies as well. What is the Lambda cold load generator? It is two files, one fairly large [sys:cold;coldut] and one medium size [sys:cold;coldld]. Mostly COLDUT implements a simple virtual memory on a disk partition. Using a disk partition instead of building the cold load in memory allows considerably more stuff to be incorporated in the cold load. This is a simplification since then there is less need for the system to run in a "primitive" mode where stuff you normally expect to be there isn't. For example, the lambda cold load has EVAL, the package sytem, READ, PRINT and lots of other stuff as well as FASLOAD all there to start. The Falcon has only FASLOAD to start, etc. In other words, it becomes feasible to put most of the stuff in the WARM and HOT file lists directly in the cold load. (The idea way to make a cold load, BTW, is to load the entire system in a separate MOBY section. That would truly avoid "primitive" modes) The lambda, of course, builds extensive list structure at cold load time which becomes part of the cold load, avoiding numerous problems, etc etc. The virtual memory system of the FLEABIT-compiled system is seriously wedged in various ways. Fixing this without breaking everything would be a royal pain. On the other hand, the virtual memory system of the lambda is perfectly adequate in principle (of course, a fair number of details need modification). It "sort of" comes for free along with the cold load generator. What needs modification to make the base lambda cold load generator suitable for the FALCON? The concern of the cold load generator is entirely with data structures. In other words it is greatly concerned with how an array is stored, but cares not what the PDL looks like, let alone a stack-group, etc. So it needs to know about simple data types like arrays, CONSes and compiled functions. The current thought (not yet a plan) is to say that the low level conventions used by the FLEABIT are for the most part OK for these simple data types. [The main exception to this I am aware of is the need to maintain back pointers to entries in functions in case the function gets superceeded. A minor exception is that the STRUCTURE datatype, etc, would be flushed and defstruct turned into arrays exactly as on the Lambda. Other suggestions for improvements that people are aware of in this area should be voiced, soon if possible, please.] An the next level of detail, converting the cold loader to the fleabit low level data storage conventions involves generating new QCOM and QDEFS files. These files serve as the authoritative, on line reference for much of the storage conventions. The hope is, the same FBIN format currently used by FLEABIT can also be loaded by the cold load generator. We have a couple of legs up on this, because it does currently know how to process QFASL files, which are basically in the same format (less several new FASL-OP codes). One significant piece of work would be required would be to put PRIMITIVE operations into the cross compiler. I guess I feel the user-level syntax used by the FLEABIT is OK, altho it would be interesting if anyone could suggest an improvement. Also required is a method to get DEFAFUNs into FBIN files. What general stages might things go thru? (1) cold loader grossly forked. lambda specific stuff flushed etc. (2) parameter files converted. [quite straightforward, providing we really do use the FLEABIT data storage conventions] (3) ability to load cross-compiled files which contain only data. (4) ability to load cross-compiled files which contain code. [this involves design of some new data structure as well as writing a medium sized piece of code, the basis of which already exists however] (5) assembling "cold-load" files (using pieces out of FLEABIT where possible) [fairly big job. Array, CONS, etc extracted, de-package-randomized, cross compiled, etc.] (6) ability to get cold load from LAMBDA disk partition into FALCON memory. (7) ability to actually start and execute cold load. (8) new debugger interface designed. Includes ability to step machine and save state. more or less at this point, we are theoretically pretty much ready to take whatever code has been processed thru to run on the fleabit system and bring it up here. In principle, the same FBIN file could be loaded into either system! [even package screwwage might not be a problem for new or LAMBDAoid code!] Note that we get a long long way into this process before we have to deal with control data structure such as pdls or stack-groups. Once we accomplish this, how does conversion to the 40 bit machine go? Pretty much, nothing having to do with this whole thing is changed. If any changes to array formats, etc are made, the corresponding changes need to be made here. Changes on the order of adding tag bits to fixnum arrays or code are trivial. One problem would be access to 40 bit disk files, etc. If we have these going by that time on the MAC, this would be a good way to test them out. Once inside the Lambda, practically everything is a bignum anyway so 32 bits versus 40 makes little difference. ==================== Datatype by datatype comparision FLEABIT vs new proposals. The general assumption above is that data representation (as opposed to control structure-representation) of the FLEABIT system is OK and can pretty much be used. This section will serve to record the blow-by-blow as we attempt to verify that assumption. [A few new issues are also being taken up here consistant with our recent decision to aim 40 bits, etc. One notable one is to see how much of a squeeze 5 bit datatypes would be.] Much of the detail of this work will be recorded in jb:k.ncold;fqcom. This file will attempt only to skim the surface. --Jim In the first stage [COLD], no symbols, etc are available at all! This causes simple looking code to fail in strange ways (clobberring NIL), etc. In the next stage [WARM] there is a completely simpleminded INTERN which involves a single OBLIST, absolutely no package structure, etc. Interning things in this way can lead to all kinds of screws later (if package inheritance fails to happen, etc etc). Finally, when the package system comes up, things have to be hacked grossly to be consistant with the multiple extraneous packages, etc which have been introduced in the FLEABIT and FLEABIT-COMPILED systems on the lambda. This obviously needs to be flushed. The Falcon has only FASLOAD to start, etc. The lambda, of course, builds extensive list structure at cold load time which becomes part of the cold load, avoiding numerous problems, etc etc. What does etc. refer to in these sentences? |||rg:The problems just mentioned previously.||| There are a number of other gross deficiencies as well. What exactly are the other gross deficiencies..... |||rg:I think enuf have been enumerated. Many lead from one to another as indicated. Because of no list structuture then packages are eventually screwwed, etc. The virtual memory system is also seriously comprimised because it has to work before lots of things exist.||| Mostly COLDUT implements a simple virtual memory on a disk partition. Using a disk partition instead of building the cold load in memory allows considerably more stuff to be incorporated in the cold load. What else does it do? |||Rg:all the stuff in WARM and HOT for starters.||| This is a simplification since then there is less need for the system to run in a "primitive" mode where stuff you normally expect to be there isn't. What is stuff in this sentence? |||RG:SYMBOLP, SPECIAL VARIABLES.|| In other words, it becomes feasible to put most of the stuff in the WARM and HOT file lists directly in the cold load. What from hot and cold can't be put in the cold load? |||rg: substantually all the stuff||| The virtual memory system of the FLEABIT-compiled system is seriously wedged in various ways. Why? |||rg: for one, the prime allocating structures are not easily accessible. They exist only in global variable type places and funny operations are required to access them. Without stopping the machine, you can not even locate them in main memory. Also the map entry has two spearate and semi-unrelated fields which interact in unclear ways. (the FRESH bit and the rest of it)||| Fixing this without breaking everything would be a royal pain. Why? |||Rg: it would fundamentally change the whole thing|| On the other hand, the virtual memory system of the lambda is perfectly adequate in principle (of course, a fair number of details need modification). What needs modification? |||RG: This is being worked out. Consult jb:k.ncold;fqcom for the current state of things||| It "sort of" comes for free along with the cold load generator. What does "sort of" mean here. |||Rg:The cold load generator also allocates memory and has mechanism for doing it.|| The current thought (not yet a plan) is to say that the low level conventions used by the FLEABIT are for the most part OK for these simple data types. [The main exception to this I am aware of is the need to maintain back pointers to entries in functions in case the function gets superceeded. A minor exception is that the STRUCTURE datatype, etc, would be flushed and defstruct turned into arrays exactly as on the Lambda. Other suggestions for improvements that people are aware of in this area should be voiced, soon if possible, please.] What exceptions are there besides this are you aware of? The hope is, the same FBIN format currently used by FLEABIT can also be loaded by the cold load generator. We have a couple of legs up on this, because it does currently know how to process QFASL files, which are basically in the same format (less several new FASL-OP codes). Why is this just a hope? What possible problems could arise? (1) cold loader grossly forked. lambda specific stuff flushed etc. Why can't the cold loader be conditionalized? (2) parameter files converted. Why can't this file be conditionalized? (6) ability to get cold load from LAMBDA disk partition into FALCON memory. Remember we must address the Macintosh issues here. (8) new debugger interface designed. Includes ability to step machine and save state. Is this the K equivalent of the LAM program. How will this interact with what Montreal is doing. Can we give them this piece of work to do? more or less at this point, we are theoretically pretty much ready to take whatever code has been processed thru to run on the fleabit system and bring it up here. Whats more and whats less and why on theoretically? What other problems might we have? Pretty much, nothing having to do with this whole thing is changed. What will have to change EXACTLY?? One problem would be access to 40 bit disk files, etc. What does etc. mean here?