Future: Streaming lexer #145

New Issue

deavmi · 2023-07-16T14:21:03+01:00

deavmi commented

2023-07-16 14:21:03 +01:00

What is this?

Re-write the lexer to stream characters in (from a string to long, me @deavmi , will adjust that to file-IO later)
- Build tokens up in this manner and when a token (or more) is built they are cached but the requested nextToken() returned
- We need caching as we want to be able to previousToken() and so on
- Implement this in compiler/lexer/kinds/streamer.d

## What is this? - [ ] Re-write the lexer to stream characters in (from a string to long, me @deavmi , will adjust that to file-IO later) * Build tokens up in this manner and when a token (or more) is built they are cached but the requested `nextToken()` returned * We need caching as we want to be able to `previousToken()` and so on * Implement this in `compiler/lexer/kinds/streamer.d`

deavmi added this to the Lexer project 2023-07-16 14:21:07 +01:00

gusmeyer was assigned by deavmi

2023-07-16 14:21:12 +01:00

deavmi added the

lexer

label 2023-07-16 14:21:15 +01:00

deavmi locked as Off-topic and limited conversation to collaborators 2023-07-16 14:21:21 +01:00

deavmi changed title from ~~Streaming lexer~~ to Future: Streaming lexer

2023-07-16 14:21:41 +01:00

deavmi added the due date 2024-08-25

2023-12-23 22:24:00 +00:00

deavmi added this to the Performance and Clarity milestone 2023-12-25 09:05:48 +00:00