mirror of https://github.com/tbklang/tlang.git
⚡ Feature: Multi-module support #157
Labels
No Label
dependency
emit
hashmapper
lebanonmapper
lexer
meta
needsfix
parser
qol
question
resolution
typing
No Milestone
No project
No Assignees
1 Participants
Notifications
Total Time Spent: 5 days 1 hour
Due Date
deavmi
5 days 1 hour
Depends on
You do not have permission to read 3 dependencies
Reference: tlang/tlang#157
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What is this?
We want to add multi-module support which means that we should be able to:
Parsershould be able to cache the information on which modules have been parsed already as to not parse them again and cycle foreverFor points 2 and 3 we should make use of the
Programtype which should contain theModule[]array.Todo
Fixes
CompilerobjectCompileras it is used on the command line, therefore we won't have to do a lot of the manual workParser
Parser'sparse(string path = "")callimport <module1>, <module2>;)Compilerinstance so we can access its utilities and most importantly itsPrograminstance (need this for managing modules)module <name>;) matchModuleobject trackingModuleas already-visitedModuleManager
ModuleManager-DDBG_MODMANsearch(...)Absolute-based path resolution ("many names for same module")Configuration
ConfigEntry()is added for these whendefaultConfig()is calledCommand-line
--pathswhich you can repeat and it tacks on to the search paths arrayAllow specifying multiple modules on the command lineChangesets
Below I show the tags one must look out for pertaining to big changes made within this branch (Note: These are in chronological order):
multimodule/changeset1debugmodemultimodule/changeset2unitteststo compileOkay, did some good stuff today. I know where I will need to continue during my free time this coming week.
The main stuff right now is coming up with smart ways to serach for available modules, then match those up to the modules wanting to be imported. Only then can we move onto the whole parsing, anti-cycle detection and so forth.
I should maybe start by first defining the search structure for a module.
Added a
ModuleManagerwhich is to aid in searching for modules and so forth.This is still being worked on.
Discussion 1 🚧
So I have a module
a.tand it has:Currently I think I am going to search the directory of the module where the current module (a.t) resides, then after that it should be based on the path or something of the name, I am not sure relative to what though. It's as if must find the most common directory and I have some ideas for that but want some input @gusmeyer
Discussion 2 🚧
Also thinking I will have to return a composite data type of file name (where it was found), probably should make that absolute when it is saved. And then the module name, that would be relative to the directory it was imported from.
Paging @gusmeyer for input. I am going ahead with this idea now as I think it may work, but it is still to be discussed as I may be missing things and your input is required.
Update 1
I have now ported over the searching code (has optional recursion):
Unit tests to follow but this looks good.
Update 2
I have it generating the names of the
ModuleEntry's correctly now it seems:I have made some good progress on module name resolution now. More is still to come...
I have added a lot now, reworking the system actually.
I must now, however, sleep 💤 .
Somethings have not yet been committed, but I have the module's now being parsed with a adhoc
LexerInterfaceand thenParserwhich are created inparseImport(string)for the sole purpose of parsing the module just read in.There is some rudimentary visitation marking but it needs to be cleaned up and made nicer and also I need to do an early call to it for the entry point module.
Done for today
Updates
DMD, the only search path added is the containing directory of the module requesting to be compiled on the command line
And here similar, I am compiling it from outside and the only directory considered on search path is the containing directory of c.t, a.k.a. niks/
Automagic 🪄 path consideration
I am now doing the following, if direct lookup within the search paths fail, then what we do is we take any module who is requesting to be imported and contains
.'s in its name and then we construct relative tacking-ons to each search path to try find it:I think it bloody works!!!!! :celebrate:
I think it is working.
Yes, looks good to me! Working on commit
41157158f241ca1cb6bde3d5305adeb610104d4a(forgot to switch over in the previous commit)I would like to merge other changes first before I move onto changing other things, especially in
DGenas that is where a lot of little things will need to be updated.I have added an addition after the entry point is parsed to add the entry. Still a work-in-progress.
Lots of things I have not yet pushed but I am working on.
Not yet pushed 🌬️
checkContainerCollision(Container c)which now can check the entities of any container and ensure they do not match the names of anyModule(s) defined within the currentProgramWe need to update the
Resolverto support aProgramand basically root itself to one of theModule, this does need some thought though in how we go about it.I need to actually do something that requires breaking in some new changes. Specifically this revolves around how we call the
resolbeBest(Container, string)function. The reason for this is that we need to be able to look at, for example, the two modules belows1 ands2which make up a singularProgram`:and...
And actually have a "current module" context we call it with such that we can easily resolve things like
xrelative to the current contextModule, else the lookup would just scan till we get first match which is wrong as these are distinct.❕We need a new dimension in the call
So I think I have a fix for this, that would look something like:
We just need to, however, fix the parenthood of expressions as that is what is causing problems with a particular test case of
cat source/tlang/testing/typecheck/simple_function_call.t(with the nestedtest()call for processingFuncCall):I need to do:
Whenever I call:
And I must modify the
parentToContainerfunction to do this for us on a per-Statementtype basis.Continueing work now...
Just got all unittests to pass 🤯
extern_test.shhad#1/bin/bashinstead of#!but fixed now.Going to continue working on the updated
parentToContainer(Container, Statement, boolean allowRecursivePainting = true)method nowAdded code which, if the
Statementincoming is aBranch, then when we extract theExpression-based condition and the body statements (Statement[]) then we must parent them But not to the incomingContainer(container) but rather to theBranchitself (to maintain its ancestry tree).Pushed some changes for the
Resolverthat were needed to make sure name generation viagenerateName(Container, Entity)worked in the newProgram-based paradigm.Some other work is still being considered for some other methods such as
resolveBest(Container, string)All resolution code has now been pushed ✅
Updates for
Programhave been pushed ✅Now that the
Resolverhas support for basically doing this, in the form of the below code ingenerateName(Container relativeTo, Entity entity):We no longer need to do this and can just call it like:
I have converted to this in all situations but one, so where it makes sense.
FunctionCallobviously needs it as it is resolving theFunctionitself.Pushed updated
Compilercode ✅All for now, will continue tonight
Working on this again now...
Pushed updated code for
Parser✅I think everything is done now in terms of moving that over! ✅
Next up: Deprecate and remove
getModule()such that we can start leaning more heavily onthis.program!Next few things
DGen
We need to make some changes to the emitter:
emitHeaderComment(string headerPhrase = "")Moduleto methods such as theseContinueing work on this now...
Got some basic fixes in, I think first we focus in the following order:
TypeCheckerDependencyCodeEmitterDGenPushed updated code for
builtins✅Pushed updated code for the
TypeChecker✅Lots of work must be done now in the
Dependencymodule as we need to have a properrootset and all, I will probably have to rework or restudy some stuff on it but it shouldn't be too bad, perhaps the ability for a clean up will thus be enforced upon me 🧠Honestly now is probably a good time to put a pause on this particular PR, and maybe I should/can pivot to a new branch solely focused on cleaning up the dependency generator as I think that would be a good thing to spend time on, looking at #41 (comment)
Going to work on #161 now then.
Fixing CI/CD pipeline...
Ah probably just build issue, nevermind :)
Will continue to work on this throughout these holidays...
Just looked into working with
gcc(or anyccfor that matter) to be able to generate static ELF libraries and also link that against a object file with a valid (extisting)mainsymbol.Placed some ideas in the new module
tlang.compiler.codegen.mapper.api🧠Been working on sorting out
DGennow. What I have started doing is to now make all theemit(...)functions take in both aFile modOut(for which the emitting must be written to) and aModule modof which specifies the currentModulebeing processed.I have also been working on new
SymbolMapperV2implementations which can be used for properly mapping the symbols.Most errors in in
DGenare now gone, but not yet committed as I will only do that once testing passes.I have also done the same for
CollidingNameException.Looks like everything compiles now (not yet unittests) so now I am going to begin to test things.
Next thing to work on is the call to the compiler, that needs some work.
More work in
DGendone on generating.oobject filesLooks like it works, for uni-modules (singular modules). Things will definitely be breaking when we move onto multi-module as we are not done yet and things still need to be done for it within the
DNodeGenerator.Working now on simplifying the
finalize()implementation inDGen. I am working on adding cleanup routines to clean up generated.cand.ofilesThrew in a
DNodewithnull(because we don't have a specialDNodeforProgramtypes yet (which notably are NOT a kind-ofStatement)).We can see everything still works as is because the DFS/linearization works as expected. The
Programpseudo DNode at the end is just ignored:You can see it ignored here:
Pushed first round of changes.
Trying to sort out unittests now, compilation errors
Compilation errors solved, runtime errors now (tests failing ⚔️)
We are getting:
For program:
Implying resolver is BROKEN!
No wait, my test update is wrong, let me rollback.
Fixed ✅.
Now a having an exception related to
DGen's call toccexiting not-nicely:Just upstreamed #162 into here, investigating quickly failing unittests
Ah it seems it was just same issue as before.
Fixing these now.
Just switched back to using an
elsefor such cases.Busy upstreaming #161 into here...
Done and unittests pass ✅
CI/CD also passing ✅
Now working on updating what the
DNodeGeneratorreturns when we callgenerate()and also how we handle this in theTypeChecker.What I am planning on doing is getting all immediate dependencies of the
Program(which would beDNodes of solelyModuletypes) and then using that as part of the per-module typechecking/code-generation.Just added:
ProgramDepNodeSo far so good, we are getting there:
We still need to do the following:
DGen/CodeEmitterTypeCheckerSome work has been done now to add
Module-based ownership ofFunctionDatafor function definitions. Still very much a work-in-progress though.Now busy with code queue work in the
TypeChecker...Going to use a mapping structure like:
Updated to this:
Almost done, just static init it seems (which needs to be done). Now the errors only appear within the
DGen.Looking at now updating the
CodeEmitterfor the code queu updates.Easy changes,
DGennextQuick finish up tonight
CodeEmitterDGenModuleTypeCheckerinitQueueneeds to be correctly generated with respectiveModuleQueueFor
CodeEmitterI need to select the correct map or something and then the correct sub-function definition, need to think a little bit about this.Actually, I think we have this is fine, we use
TypeCheckerto get aInstruction[][string]map which we select byModule, then we get afuncName -> InstructionQueuemap of function definition groupings.This is fine. Then we loop over each and there we must select the queue.
Nah wait, we got a problem. Undoing changes, we use methods that get function names but these are not yet using
Module, I need to update this. Maybe no undoing is needed, just changing it will make IDE show where updates are neededCompiles, unittests pass - holy kak!
We are making money moves
(Also if @rany is seeintg this, then mashallah)
Parserstuff now just needs to be sorted (it causes multiple compilations of entry module), hence linker fails, which makes sense:All changes that got us to this point have now been committed ✅
I now need to work on updating the
Resolverto have a generic reslver method that takes in basically a predicate. I should probably do this on another branch maybe.See #166
Got the latest
Resolvercode in here now. Now I can begin work on the emitting (forDGen) of all public (should beextern) entitiesI actually need a sort of find-all mode, needs a change to the
ResolverthereforeGot this now
Working on the
ModuleExternSetstuff nowWill take a look at this again this weekend.
Next fx is for when we run:
We get:
Just added:
To the
Programindata.d:Updated assertion in
resolveUp(..., ...):assert(cast(Entity) currentContainer || cast(Program)currentContainer);Feel as though I need seperate handling in this case.
Not the only problem, anther needs to properly root when a module name is encountered.
Okay problem is name being queried is not jhow can I say. Basically the functoin s being lked up on its name. Determined to be as so in the
FuncCallInstrwhen looked up usingb.doThingin typecheck/codegen. So we need to maybe adjust the context or do something like that.Better yet, we should leave the name in the
FuncCallInstrmaybe?Wait, we do that already, maybe generate full name. Let me take a look.
Eays fix by using
funcCall.getName()which has full AST name from the expression itself. let's use that and not thegetname()of theFunctionlooked up from there.This seems to fix it, gets to the actual
gcccall now, good prog.This is GOOD! 🎊 :
Okay, now looks like symbol mapper stuff is what we need to sort out now.
Have to check some stuff now, but dinner time! 🍷
Symbol mapping stuff still being worked on and tested
Sorted it out, sticking to one
ScopeTypefor now. Now the symbols match and the C compiler is happy.Going to continue on this today.
Trying to figure out now why I have a certain unittest in the
Parserfailing:Conirmed that it is the unittest in
parsing/core.dwith themyModuleFrwhich is crashing:Okay the problem seems to be when searching for an entity which does not exist. We know that nthere is nothing named
innerinside themyClass2(the one declared at the module-level):But this crashes the
Resolverit seems. We need to therefore fix that.Okay I hjave an idea what the problem is, It is within
resolveUp(Container, Predicate)... looking into it nowThis is the fix, and now it seems to work:
Time to write a commit message, clean up the print statements I now have scattered everywhere and so on
Pushed fixes for that. Builds will still be failing because of things not yet pushed (unrelated to the
data.dandresolution.dstuff).Pushed the fix for the
HashMapper✅Also includes fixed unittest for it
Pushed:
CodeEmitter
SymbolMapperV2for the mapperCompiler
HashMapperandLebanonMapper.SymbolmapperV2APINow what remains is just updates to
DGenand theTypeChecker... I will clean these up and push when I can.Just pushed:
DGen
SymbolMapperV2GLOBALScopeType. n this case have added comments above each mapping request to reconsider which scope type should be used on a case-by-case basis.emitStdint(..., ...)prior toemitExterns(..., ...)because the latter uses types defined in the former (think of well, theuint32_ttypes for example)ModuleExternSetwhich holds aModuleand its respetcive publically accessibleFunction(s) andVariable(s)generateExternsForModule(Module mod)which generates aModuleExternSetfor the givenModuleemitExterns(File modOut, Module mod)to now determine allModule[]except the incomingmoditself and then generateModuleExternSet()s for each. These are then looped through and extern statements are generated for eachModuleExternSet'spubFns()and (soon) we will also add support for thepubVars()` of eachJust the fix for
TypeCheckerthat I need to addJust pushed:
TypeChecker
DNodewhich contains aStatementof which is aFunctionCallwe now will generate the correspondingFuncCallInstrusing the name that is refered to in theFunctionCall(i.e. usinggetname()on theFunctionCall) INSTEAD of using thegetName()method from theFunctionlooked up CVIA theFunctionCall'sgetName()method. This ensures that the full path expression is passed into theFuncCallInstr. Remember it is looked up later in the emitter but having this name fixed a lto of problems. i.e. having ab.doThing()call in moduleanow works. Previously it would look up justdoThng()as that is all that was all whic was saved within theFuncCallInstr.Instructionwhic refers to some entity using a name, save-what-you-see. Don't eaergly lookup stuff and re-generate names because the name y7ou see is normally syntactically correct from the user.Compilation is working and tested semantics is all good with something like
simple_functions.t. Unittests also pass.Waiting on Github CI/CD....
CI/CD passed ✅
Also to note coverage is back up where the coverage tracker expected it to be (for an improvement to have been marked or at least recovered to the highest it has tracked yet).
Parserfixes are what is now next.When we are closer to being done e should also document everything to some degree. I definately think documenting the
Resolverwith its new updates would be worth whileThings to do for
ResolverProgramto try trigger any buggy codeWorking on the former now and there is already a bad implementation of
resolveUp(..., ...)forProgram-rooted rsearches. This was never tested so am glad I am doing it now. can already see some of the problems with it. If it haspath.length==1then it tries to find firstly if it is the nqame of aMdoulehowever when not matching it returnsnull. It should fall through and see what else is named with the given name, because that is not right (obviously).Look, actually it is odd to want to resolve stuff in this manner to be qute honest as the results will not be that predicatble. It should probably return
nulland that probably makes the most sense actually.Need to fix is descendant,
assert(tc.getResolver().isDescendant(cast(Container)program, myModule));failsThis is fine now it seems.
Now fixing
genwerateNameBest(Entity)with a sim,ilar sort of issue. May rewrite it, I don't like the while l,oop instead of a dfo-while loop.Such an easy fix by using
findContainerOfType(..., ...)UIp to 92% coverage on the resolver now....
Pushed now ✅
Everything is passing ✅
Gustav idea for unique single parse for any given module import/visitation startergy:
Did some clean up
Will do this tomorrow
Slight variation on this, we can actually do it from within the call to
parse()when it has already constructed an emptyModuleand we also get the main thing, the ability to know the module's name.In any case this is way cleaner and what I wanted from the beginning, therefore I shall add it as so. The way we will do this is with a default parameter to
parse(string)asparse(string, bool isEntrypoint = false). And the only time we shall setisEntrypointtotrueis when we callparser.parse()from theCompiler'sdoParse()code (this is the entry point to the entire parsing process anyways).This is done and it works! ✅ 🎊
Local tests pass and CI/CD
Semantic local test passes too, will update the GitHub CI test to check for exit code to ensure automatic testing of it
Writing documentation now
Todo
Will continue working on docs after I shower 🚿
Both updating in-code docs whilst I type up the actual docs - win-win.
Added documentation on how the
ModuleManagersearches for modules, that is almost done.Also fixed up some inefficient parts of the code for that.
Should probs get to that testing now but I just wanna finish up the docs real quick.
The
ModuleManagerand theModMasnmodule (at large) is all code-documented now.I also believe that the docs written are all done and now available in a file named
small_doc_mopdules.md.Ah, it seems that there is a part on
ProgeramandModule, I feel that should be done earlier ina secrtion on theResolver. Will begin work on that now.Removed those parts, they will be done in the section on resolution.
ModMan docs are done and moved into the docs repo as well
Will do resolver tomorrow, time to sleep now 💤
Resolution docs are being worked on now
This is progressing, only a few more mthods left.
There are also code improvements coming along with this.
Good Friday
The name generation methods are documented now and there are some code changes as well which will be commited soon ☑️
The next set of methods requiering overview and documentation are those relating to pertforming resolution.
Just pushed:
Resolver
generateNameBest(Entity entity)now relies ongenerateName(Container, Entity)'s special handling ofProgram's in order to achieve its goal.generateName(Container, Entity)generateName_Internal(...)is nowgenerateName0(...)Done for now, more fixes to come code wise and also finish up resolution docs
Doing some more documentation work now
Done for now, will continue later
Cleaning up
resolveBest(..., ...)nowDocumentation for the resolver is basicallly done ✅
The resolver code is also done being reviewed ✅
Parsing (unittests)
a.t/b.tandniks/c.tin a unittest itselfPipelines
a.t(multi-module) testLooks like automated test working as expected, took some time to get it right in the GitHub pipelines file
d.yml✅Parser
doImport(string)now usedoImports(string[])parseImport()now supports multi-line importsMulti-line import support added ✅
Working on this now, will make it disabled by default first though - just for now because it breaks unittests if enabledd by default
There are some unavoidable unittests that need fixing too now.
Cleaned out two unused methods in
modman.dSorted out.
The comments tests are disabled for now because we don't yet associate comments with AST nodes (which is needed therefore). 🐤
But other tests was fixed which now uses a
Compileras intended ✅This is now done ✅
Last few things
--pathsLast bugs 🐞
As we can see below when we added
/usr/libto the search path then it errored on the first try of not being able to find aniks/in there (presumably forniks/c.t):This will actually require a rework of some of the code in the module manager, so I need to work on this today.
Wait no it is fine me thinks?
Actually no, we do need to do this it seems. Just been busy sorting out adding support for
--pathson the command-line.Found the error. It was the
findAllTFilesShallow(string)which calleddirEntries(string)without firstly checking if such a path was valid on the file system or, if so, if it even is a directoryThis bug has been fixed ✅
I lie, it kinda is fine but validation needs the same fix
Yes, so find shallow is a silent failure as it needed to check if something could exist before enumerating files in such a directory, validation - rather - has to do with the starting path - if an initial path passed on the
--pathscommand-line flag is valid to start with.Fixed ✅
Everything is done ✅
Not just yet done, few more things.
Done ✅
Just finished the parser docs ✅
Everything here is now done ✅
Okay there are a few more things to work on maybe,.... but these are low priority, so I can stop working on this PR for now at least
Last few things
DGento emit extern variables (not just functions)extern-like statements must become automaticallyprivate. This will stop them from being accessible outside to other modules which would cause them to get double externed. I mean, is that a problem. We could maybe handle it. But maybe not for now? #168vardec_varass_dependencyDgen
generateSignature_Variable(Variable var)which generates JUST a variable signature (with symbol mapping)emitExterns(File modOut, Module mod)now also generates extern statements for global variablesWe should make all
extern evar/efuncstatements simplyprivateso that nobody can have them as something to export.Busy sorting out merge conflicts now...
All synced up, pushed ✅
Local unittests pass ✅
Cleaned up symbol mapping modules.
With regards to the second item in this list I think it actually may not require much work to get right. Look when we generate the externs for module import system we do so by placing
extern <type> <name>, where<name>is theEntity's name regardless of whether itisExternal()or not.Hence it should just work. The only part we must worry about is that we must NOT symbol map the
Entity's (Variables) which are extern.So basically this:
Okay but wait, this is fine and all but technically if we request a signature for a
variablethen it should also place theexternin front of it.This is where the problemo lies. Who's resposibility. I say this because the function siganture generator does this and therefore, to stay consistent, so should the
Variable-based one.So:
And now we shall run into expected problems that should be fixed in the
ModuleExternStmtsthing.Heheh:
Okay, but now that it is fixed we could also then do checks in the
ModExternSttmsdthat yanks out theexternpart if is knows the entity the signature is requested for isextern.This would then make the following
evar/efuncs:into:
The checks are therefore basically like this:
Looks like this solution works. ✅
Second item is now done, making this list now entirely done ✅
Just last few things that I want to document and then I will move this into the queue pending a merge...
Done ✅
Just finishing up parser docs now...
Still working on this...
Parsing
Okay, pretty much happy with this now
Merging...
Merged ✅
Merging docs...
Done ✅