Lexer interface #105

New Issue

deavmi · 2023-03-08T14:23:51Z

deavmi commented

2023-03-08 14:23:51 +00:00

What is this?

The point of this feature request is to declare a set of files that hold Token, LexerInterface and then a generalized LexerInterface. The interface should declare the general structure that a parser must adhere to in order to be used, namely:

There must be a getCurrentToken() method which returns a Token
Implement nextToken(), previousToken()
Cursor position set/get with setCursor(ulong) and getCursor()
There must be a hasTokens() to check if there are more tokens to come considering the current cursor position
An exhaustive getTokens()
A getLine() and getColumn() to know where in the source text we are

# What is this? The point of this feature request is to declare a set of files that hold `Token`, `LexerInterface` and then a generalized `LexerInterface`. The interface should declare the general structure that a parser must adhere to in order to be used, namely: 1. There must be a `getCurrentToken()` method which returns a `Token` 2. Implement `nextToken()`, `previousToken()` 3. Cursor position set/get with `setCursor(ulong)` and `getCursor()` 4. There must be a `hasTokens()` to check if there are more tokens to come considering the current cursor position 5. An exhaustive `getTokens()` 6. A `getLine()` and `getColumn()` to know where in the source text we are

deavmi added a new dependency 2023-03-08 14:23:59 +00:00

#104 Lexer re-work

deavmi referenced this issue

2023-03-08 14:24:20 +00:00

Lexer re-work #104

Ghost was assigned by deavmi

2023-03-08 14:24:26 +00:00

deavmi added the

parser

label 2023-03-08 14:24:38 +00:00

deavmi changed title from ~~Parser interface~~ to Lexer interface

2023-03-08 14:25:02 +00:00

deavmi added

qol

lexer

and removed

parser

labels 2023-03-08 14:25:07 +00:00

deavmi referenced this issue

2023-03-08 14:28:49 +00:00

Streaming lexer #106

deavmi added a new dependency 2023-03-08 14:29:17 +00:00

#106 Streaming lexer

deavmi commented

2023-03-08 14:30:26 +00:00

🔐️ Locking for now till I stabilize stuff and onboard @rany

deavmi locked and limited conversation to collaborators 2023-03-08 14:30:39 +00:00

deavmi added this to the Clean ups milestone 2023-03-08 14:32:19 +00:00

deavmi added this to the Parser project 2023-03-08 14:32:50 +00:00

deavmi added reference feature/lexer_interface

2023-04-20 11:05:13 +01:00

deavmi commented

2023-04-20 11:08:35 +01:00

I have pushed a definition for LexerInterface now

I have pushed a definition for `LexerInterface` now

deavmi commented

2023-04-20 13:05:05 +01:00

Todo 📝

Add getTokens() to the interface API
- Perhaps we should define a getTokens(), which exhaustively calls getTokens()till it cannot anymore. Useful for unit tests where we need to compare tokens.
getLine() and getColumn()

## Todo 📝 - [x] Add `getTokens()` to the interface API * Perhaps we should define a `getTokens()`, which exhaustively calls `getTokens()`till it cannot anymore. Useful for unit tests where we need to compare tokens. - [x] `getLine()` and `getColumn()`

deavmi commented

2023-04-20 13:31:41 +01:00

Questions 🤔

getTokens() is common code
- It could be argued that we may want this to be an abstract class rather than an interface in the case of getTokens(), as we could simply call it exhaustively ourselves. It would be common code
- Solution, we don't even need a class, we can use a ~default~ final interface method for this, mmmh, however maybe we do? We would need to reset the cursor each time - it is doable but we can see - honestly who would call getTokens() in places other than unit tests and also more than once? It will not change.

## Questions 🤔 - [ ] `getTokens()` is common code * It could be argued that we may want this to be an `abstract class` rather than an `interface` in the case of `getTokens()`, as we could simply call it exhaustively ourselves. It would be common code * **Solution**, we don't even need a class, we can use a ~`default`~ `final` interface method for this, mmmh, however maybe we do? We would need to reset the cursor each time - it is doable but we can see - honestly who would call `getTokens()` in places other than unit tests and also more than once? It will not change.

deavmi self-assigned this 2023-04-20 13:34:45 +01:00

Ghost was unassigned by deavmi

2023-04-20 13:34:46 +01:00

deavmi referenced this issue

2023-04-20 13:53:05 +01:00

Lexer re-work #104

deavmi commented

2023-04-20 13:58:30 +01:00

Questions 🤔

getTokens() is common code

It could be argued that we may want this to be an abstract class rather than an interface in the case of getTokens(), as we could simply call it exhaustively ourselves. It would be common code

Solution, we don't even need a class, we can use a ~default~ final interface method for this, mmmh, however maybe we do? We would need to reset the cursor each time - it is doable but we can see - honestly who would call getTokens() in places other than unit tests and also more than once? It will not change.

Currently doing this crashes with out of bounds, need to check how basic.d works.

> ## Questions 🤔 > > - [ ] `getTokens()` is common code > * It could be argued that we may want this to be an `abstract class` rather than an `interface` in the case of `getTokens()`, as we could simply call it exhaustively ourselves. It would be common code > * **Solution**, we don't even need a class, we can use a ~`default`~ `final` interface method for this, mmmh, however maybe we do? We would need to reset the cursor each time - it is doable but we can see - honestly who would call `getTokens()` in places other than unit tests and also more than once? It will not change. Currently doing this crashes with out of bounds, need to check how `basic.d` works.

deavmi removed a dependency 2023-04-20 14:12:30 +01:00

#104 Lexer re-work

deavmi commented

2023-07-08 15:27:11 +01:00

I want to start work on this soon :). Maybe once I finish some of the type checking stuff up.

deavmi referenced this issue

2023-07-08 18:00:44 +01:00

🧹️🧼️ Cleanup: Clean up series #139

deavmi changed reference from ~~feature/lexer_interface~~ to feature/clean_lexer_interface

2023-07-08 18:03:15 +01:00

deavmi started working 2023-07-08 19:10:30 +01:00

deavmi added the due date 2023-07-10