deavmi commented

Owner

What is this?

The point of this feature request is to work on re-working the lexer with a few stages and in order of how they should be completed:

Generalize the interface a lexer should export using a new interface (see issue #105)
- This should be in the lexer/ directory
- Should be named LexerInterface
- Should have common methods like nextToken(), previousToken(), getCurrentToken() etc
- Common types like Token and LexerException should be moved into a general directory alongside the LexerInterface module
Take the current lexer and make it adhere to the LexerInterface
Start work on the streaming lexer (see #106)

# What is this? The point of this feature request is to work on re-working the lexer with a few stages and in order of how they should be completed: - [x] Generalize the interface a lexer should export using a new `interface` (see issue #105) * This should be in the `lexer/` directory * Should be named `LexerInterface` * Should have common methods like `nextToken()`, `previousToken()`, `getCurrentToken()` etc * Common types like `Token` and `LexerException` should be moved into a general directory alongside the `LexerInterface` module - [x] Take the current lexer and make it adhere to the `LexerInterface` - [x] Start work on the streaming lexer (see #106)

Ghost was assigned by deavmi

2023-03-08 14:22:06 +00:00

deavmi added a new dependency 2023-03-08 14:23:59 +00:00

#105 Lexer interface

deavmi changed title from ~~Parser re-work~~ to Lexer re-work

2023-03-08 14:24:45 +00:00

deavmi added the

lexer

qol

labels 2023-03-08 14:24:59 +00:00

deavmi added a new dependency 2023-03-08 14:29:21 +00:00

#106 Streaming lexer

deavmi commented

2023-03-08 14:30:15 +00:00

Author

Owner

🔐️ Locking for now till I stabilize stuff and onboard @rany

deavmi locked and limited conversation to collaborators 2023-03-08 14:30:47 +00:00

deavmi added this to the Clean ups milestone 2023-03-08 14:32:15 +00:00

deavmi added this to the Parser project 2023-03-08 14:32:52 +00:00

deavmi added reference feature/lexer_interface

2023-04-20 11:08:44 +01:00

deavmi commented

2023-04-20 13:02:41 +01:00

Author

Owner

Work has begun on translating the current lexer over to the LexerInterface API...

It passes unit tests, test cases in CI/CD also pass ✅

Work has begun on translating the current lexer over to the `LexerInterface` API... It passes unit tests, test cases in CI/CD also pass ✅

deavmi commented

2023-04-20 13:02:45 +01:00

Author

Owner

Todo

Move over token management from Parser to Lexer in core.d
Move Lexer from core.d to BasicLexer in basic.d
make core.d import (publically) tokens.d, exceptions.d etc.
Make BasicLexer in basics.d import tlang.compiler.lexer.core

# Todo - [x] Move over token management from `Parser` to `Lexer` in `core.d` - [x] Move `Lexer` from `core.d` to `BasicLexer` in `basic.d` - [x] make `core.d` import (publically) `tokens.d`, `exceptions.d` etc. - [x] Make `BasicLexer` in `basics.d` import `tlang.compiler.lexer.core`

deavmi self-assigned this 2023-04-20 13:34:41 +01:00

Ghost was unassigned by deavmi

2023-04-20 13:34:42 +01:00

deavmi commented

2023-04-20 13:53:05 +01:00

Author

Owner

What is this?

The point of this feature request is to work on re-working the lexer with a few stages and in order of how they should be completed:

Generalize the interface a lexer should export using a new interface (see issue #105)

This should be in the lexer/ directory

Should be named LexerInterface

Should have common methods like nextToken(), previousToken(), getCurrentToken() etc

Common types like Token and LexerException should be moved into a general directory alongside the LexerInterface module

Take the current lexer and make it adhere to the LexerInterface

Start work on the streaming lexer (see #106)

All done, and I have created an issue (#106) for @rany to work on when we stabalise hopefully by October ish.

API all there.

As for the current lexer, I have transformed it over to th LexerInterface API and it works (unit tests and test cases CI are good ✅ )

> # What is this? > > The point of this feature request is to work on re-working the lexer with a few stages and in order of how they should be completed: > > - [x] Generalize the interface a lexer should export using a new `interface` (see issue #105) > * This should be in the `lexer/` directory > * Should be named `LexerInterface` > * Should have common methods like `nextToken()`, `previousToken()`, `getCurrentToken()` etc > * Common types like `Token` and `LexerException` should be moved into a general directory alongside the `LexerInterface` module > - [x] Take the current lexer and make it adhere to the `LexerInterface` > - [ ] Start work on the streaming lexer (see #106) All done, and I have created an issue (#106) for @rany to work on when we stabalise hopefully by October ish. API all there. As for the **current lexer**, I have transformed it over to th `LexerInterface` API and it works (unit tests and test cases CI are good ✅ )

deavmi removed a dependency 2023-04-20 13:53:24 +01:00

#106 Streaming lexer

deavmi added the due date 2023-04-20

2023-04-20 13:53:37 +01:00

deavmi removed a dependency 2023-04-20 14:12:30 +01:00

#105 Lexer interface

deavmi closed this issue

2023-04-20 14:12:34 +01:00

Lexer re-work #104

What is this?

Todo

What is this?