tlang/tlang
tlang
/
tlang
mirror of https://github.com/tbklang/tlang.git
2
0
Fork 0
Code Issues 41 Projects 5 Releases Wiki Activity

⚡ Feature: Lexer improvements #144

New Issue
Closed
opened 2023-07-14 16:46:19 +01:00 by deavmi · 25 comments
deavmi commented 2023-07-14 16:46:19 +01:00
Owner
Copy Link

What is this?

We have a Grammar and I want to ensure that the lexer builds tokens accordingly and also figure out what is missing.

This leaves us with two main (immediate goals):

  • Unit tests for the current BasicLexer
    • Adding more to try and break it and more complicated tests
    • Adding test cases to increase code coverage
  • Actually fixing the lexer
    • In cases where is is broken
      Or it does Not yet implement everything stipulated in the grammar
    • Finding what the lexer does but grammar does not yet document and updating the docs accordingly
  • Adding tab support as per #121
  • Numerical encoders 21UL etc.
    • @deavmi and @gusmeyer should discuss this
    • Entails usage of isNumericalEncoder(char character), isNumericalEncoder_Size(char character) and isNumericalEncoder_Signage(char character)
  • Implement isValidEscape_String(char)
    • Might need more \\ things etc required
    • We should make a list of supported excape sequences we want, add support for them and so on
    • String support
    • Character support (like '\n')
  • Enum for character constants frequently used
  • Dotted handling edge cases new A()\t.\tl . p.p;
  • Bug: The lexer currently allows spaces/tabs around a standalone '.' or with spaces on either side, which should no tbe the case.
  • Investigate #60
  • Support for comments #84
  • Support for 2_100
    • This should become a token [2100]
  • Float valid-splitters function
    • Determine valid splitters for float
  • Once done speak to @deavmi as you need to document the code as well as in the documentation/manual
  • Move isSplitter(char), isOperator(char) to core/lexer.d @deavmi
    • Moved
    • Old ones to use migrated ones
## What is this? We have a [Grammar](https://deavmi.assigned.network/projects/tlang/language/31-grammar/) and I want to ensure that the lexer builds tokens accordingly and also figure out what is missing. This leaves us with two main (immediate goals): - [x] Unit tests for the current `BasicLexer` * Adding more to try and break it and more complicated tests * Adding test cases to increase code coverage - [x] Actually fixing the lexer * In cases where is is **broken** Or it does **Not yet** implement everything stipulated in the grammar * Finding what the lexer does but grammar does not yet document and updating the docs accordingly - [x] Adding tab support as per #121 - [x] Numerical encoders `21UL` etc. * @deavmi and @gusmeyer should discuss this * Entails usage of `isNumericalEncoder(char character)`, `isNumericalEncoder_Size(char character)` and `isNumericalEncoder_Signage(char character)` - [x] Implement `isValidEscape_String(char)` * Might need more `\\` things etc required * We should make a list of supported excape sequences we want, add support for them and so on - [x] String support - [x] Character support (like `'\n'`) - [x] Enum for character constants frequently used - [x] Dotted handling edge cases `new A()\t.\tl . p.p; ` - [x] Bug: The lexer currently allows spaces/tabs around a standalone '.' or with spaces on either side, which should no tbe the case. - [x] Investigate #60 - [x] Support for comments #84 - [x] Support for `2_100` * This should become a token `[2100]` - [x] Float valid-splitters function * Determine valid splitters for float - [x] Once done speak to @deavmi as you need to document the code as well as in the documentation/manual - [x] Move `isSplitter(char)`, `isOperator(char)` to `core/lexer.d` @deavmi - [x] Moved - [x] Old ones to use migrated ones
gusmeyer was assigned by deavmi 2023-07-14 16:46:46 +01:00
deavmi added the
lexer
 label 2023-07-14 16:46:52 +01:00
deavmi added this to the Lexer project 2023-07-14 16:46:56 +01:00
deavmi added this to the Clean ups milestone 2023-07-14 16:47:08 +01:00
deavmi added reference feature/lexer_improvements 2023-07-14 16:57:57 +01:00
deavmi modified the milestone from Clean ups to Basics 2023-07-14 16:58:55 +01:00
deavmi added the due date 2023-07-30 2023-07-14 16:59:06 +01:00
deavmi added a new dependency 2023-07-16 13:59:19 +01:00
#121 🐞️ Bugfix: Tabs in lexer: Not handled
deavmi started working 2023-07-16 14:39:49 +01:00
deavmi stopped working 2023-07-16 14:40:09 +01:00
20 seconds
gusmeyer started working 2023-07-16 14:44:07 +01:00
gusmeyer stopped working 2023-07-16 15:18:30 +01:00
34 minutes 22 seconds
gusmeyer started working 2023-07-16 15:27:24 +01:00
gusmeyer stopped working 2023-07-16 15:36:41 +01:00
9 minutes 17 seconds
gusmeyer started working 2023-07-16 16:11:35 +01:00
deavmi commented 2023-07-16 16:45:58 +01:00
Author
Owner
Copy Link

Minutes

Some test notes:

1.5.5

[1.5, .]

1.5.a

[1.5, ., a]

l. 5
l. a

[]

"l .5"

"dj;"

[dj, ;]

"dj
";

[dj]



").x" ").5" ") .5"
### Minutes Some test notes: ``` 1.5.5 [1.5, .] 1.5.a [1.5, ., a] l. 5 l. a [] "l .5" "dj;" [dj, ;] "dj "; [dj] ").x" ").5" ") .5" ```
gusmeyer stopped working 2023-07-16 17:49:49 +01:00
1 hour 38 minutes
deavmi commented 2023-07-16 19:02:53 +01:00
Author
Owner
Copy Link

Documentation

  • Document the fucntions in lexer (a lot of missing)
  • Update docs in book file(just ask me when you get to this stage and I can show you how)
## Documentation - [x] Document the fucntions in lexer (a lot of missing) - [x] Update docs in book file(just ask me when you get to this stage and I can show you how)
deavmi added a new dependency 2023-07-16 19:43:19 +01:00
#60 Lexer stuff
deavmi commented 2023-07-16 19:43:46 +01:00
Author
Owner
Copy Link

Updated the TODOs @gusmeyer

Added a TODO for #60 (this needs some discussing for you and me so we can leave this till later or whenever you think we should discuss it)

Updated the TODOs @gusmeyer Added a TODO for #60 (this needs some discussing for you and me so we can leave this till later or whenever you think we should discuss it)
deavmi added a new dependency 2023-07-16 19:45:58 +01:00
#84 Comment support
deavmi started working 2023-07-20 15:01:46 +01:00
deavmi stopped working 2023-07-20 15:02:32 +01:00
46 seconds
deavmi added spent time 2023-07-20 16:14:00 +01:00
2 minutes
deavmi added spent time 2023-07-22 19:35:32 +01:00
2 minutes
deavmi started working 2023-07-22 19:35:34 +01:00
deavmi commented 2023-07-22 19:36:12 +01:00
Author
Owner
Copy Link

Updated Parser such that it now tests the call to parseComment() from within parseStatement() (and not just parse()) ✅

Updated `Parser` such that it now tests the call to `parseComment()` from within `parseStatement()` (and not **just** `parse()`) ✅
deavmi stopped working 2023-07-22 19:36:15 +01:00
41 seconds
deavmi added spent time 2023-07-22 20:05:25 +01:00
10 minutes
deavmi started working 2023-07-22 20:05:28 +01:00
deavmi commented 2023-07-22 20:06:10 +01:00
Author
Owner
Copy Link

Updated Parser such that it now tests the call to parseComment() from within parseStatement() (and not just parse()) ✅

The comments tests (in the Parser) now use Gustav's BasicLexer, and they all pass ✅

> Updated `Parser` such that it now tests the call to `parseComment()` from within `parseStatement()` (and not **just** `parse()`) ✅ The comments tests (in the `Parser`) now use Gustav's `BasicLexer`, and they all pass ✅
deavmi stopped working 2023-07-22 20:06:13 +01:00
45 seconds
deavmi commented 2023-07-22 20:18:59 +01:00
Author
Owner
Copy Link

@gusmeyer We need to update the grammar docs ❗

@gusmeyer We need to update the grammar docs ❗
deavmi started working 2023-08-10 18:56:03 +01:00
deavmi commented 2023-08-10 18:56:17 +01:00
Author
Owner
Copy Link

Bumping due date to end of August

Bumping due date to end of August
deavmi modified the due date from 2023-07-30 to 2023-08-30 2023-08-10 18:56:27 +01:00
deavmi commented 2023-08-10 18:56:58 +01:00
Author
Owner
Copy Link

Merging vardec_varass_dependency into here as #115 is now completed and should solve CI/CD failing...

Merging `vardec_varass_dependency` into here as #115 is now completed and should solve CI/CD failing...
deavmi commented 2023-08-10 18:58:34 +01:00
Author
Owner
Copy Link

Merging vardec_varass_dependency into here as #115 is now completed and should solve CI/CD failing...

Merged, awaiting CI/CD...

> Merging `vardec_varass_dependency` into here as #115 is now completed and should solve CI/CD failing... Merged, awaiting CI/CD...
deavmi commented 2023-08-10 19:01:49 +01:00
Author
Owner
Copy Link

Merging vardec_varass_dependency into here as #115 is now completed and should solve CI/CD failing...

Merged, awaiting CI/CD...

CI/CD passed ✅

> > Merging `vardec_varass_dependency` into here as #115 is now completed and should solve CI/CD failing... > > Merged, awaiting CI/CD... CI/CD passed ✅
deavmi stopped working 2023-08-10 19:01:54 +01:00
5 minutes 51 seconds
deavmi commented 2023-08-16 10:04:29 +01:00
Author
Owner
Copy Link

Branch is synced up ✅

Branch is synced up ✅
deavmi changed title from Lexer improvements to ⚡ Feature: Lexer improvements 2023-08-17 08:06:39 +01:00
deavmi started working 2023-08-27 11:38:25 +01:00
deavmi stopped working 2023-08-27 11:50:18 +01:00
11 minutes 53 seconds
deavmi started working 2023-08-27 11:59:15 +01:00
deavmi commented 2023-08-27 12:02:10 +01:00
Author
Owner
Copy Link

Update with the latest base code from vardec_varass_dependency (unit tests now will cause more code to be covered) 👍

Update with the latest base code from `vardec_varass_dependency` (unit tests now will cause more code to be covered) 👍
deavmi stopped working 2023-08-27 12:02:14 +01:00
3 minutes
deavmi added a new dependency 2023-11-06 09:38:59 +00:00
#157 ⚡ Feature: Multi-module support
deavmi modified the due date from 2023-08-30 to 2023-12-06 2023-12-06 11:47:22 +00:00
deavmi modified the due date from 2023-12-06 to 2023-12-07 2023-12-06 11:47:29 +00:00
deavmi commented 2023-12-06 13:11:13 +00:00
Author
Owner
Copy Link

Gustav is working on this now, escape sequences for characters is added but not YET pushed! ❗

Gustav is working on this now, escape sequences for characters is added **but not YET pushed**! ❗
deavmi modified the due date from 2023-12-07 to 2023-12-31 2023-12-10 10:14:30 +00:00
deavmi started working 2023-12-23 18:41:23 +00:00
deavmi stopped working 2023-12-23 18:47:31 +00:00
6 minutes 7 seconds
deavmi started working 2023-12-23 18:51:18 +00:00
deavmi stopped working 2023-12-23 18:54:00 +00:00
2 minutes 42 seconds
deavmi added spent time 2023-12-23 20:37:18 +00:00
1 minute
deavmi started working 2023-12-23 20:37:43 +00:00
deavmi stopped working 2023-12-23 20:37:50 +00:00
7 seconds
deavmi started working 2023-12-23 20:38:15 +00:00
deavmi stopped working 2023-12-23 20:41:25 +00:00
3 minutes 10 seconds
deavmi commented 2023-12-23 22:16:47 +00:00
Author
Owner
Copy Link

This is all done now and can be merged ✅

This is all done now and can be merged ✅
deavmi started working 2023-12-23 22:30:39 +00:00
deavmi stopped working 2023-12-23 22:30:56 +00:00
17 seconds
deavmi added spent time 2023-12-24 12:48:41 +00:00
5 minutes
deavmi started working 2023-12-24 12:48:44 +00:00
deavmi self-assigned this 2023-12-24 12:50:27 +00:00
deavmi commented 2023-12-24 13:10:29 +00:00
Author
Owner
Copy Link

I am working on documentation now and also moving out common methods into other modules (along with documenting them).

I am working on documentation now and also moving out common methods into other modules (along with documenting them).
deavmi stopped working 2023-12-24 13:37:52 +00:00
49 minutes 7 seconds
deavmi started working 2023-12-24 13:38:04 +00:00
deavmi commented 2023-12-24 13:41:08 +00:00
Author
Owner
Copy Link

What is this?

We have a Grammar and I want to ensure that the lexer builds tokens accordingly and also figure out what is missing.

This leaves us with two main (immediate goals):

  • Unit tests for the current BasicLexer
    • Adding more to try and break it and more complicated tests
      • Adding test cases to increase code coverage
  • Actually fixing the lexer
    • In cases where is is broken
      Or it does Not yet implement everything stipulated in the grammar
      • Finding what the lexer does but grammar does not yet document and updating the docs accordingly
  • Adding tab support as per #121
  • Numerical encoders 21UL etc.
    • @deavmi and @gusmeyer should discuss this
      • Entails usage of isNumericalEncoder(char character), isNumericalEncoder_Size(char character) and isNumericalEncoder_Signage(char character)
  • Implement isValidEscape_String(char)
    • Might need more \\ things etc required
    • We should make a list of supported excape sequences we want, add support for them and so on
    • String support
    • Character support (like '\n')
  • Enum for character constants frequently used
  • Dotted handling edge cases new A()\t.\tl . p.p;
  • Bug: The lexer currently allows spaces/tabs around a standalone '.' or with spaces on either side, which should no tbe the case.
  • Investigate #60
  • Support for comments #84
  • Support for 2_100
    • This should become a token [2100]
  • Float valid-splitters function
    • Determine valid splitters for float
  • Once done speak to @deavmi as you need to document the code as well as in the documentation/manual
  • Move isSplitter(char), isOperator(char) to core/lexer.d @deavmi
    • Moved
    • Old ones to use migrated ones

Migrated all methods that were common char-based ones

> ## What is this? > > We have a [Grammar](https://deavmi.assigned.network/projects/tlang/language/31-grammar/) and I want to ensure that the lexer builds tokens accordingly and also figure out what is missing. > > This leaves us with two main (immediate goals): > > - [x] Unit tests for the current `BasicLexer` > * Adding more to try and break it and more complicated tests > * Adding test cases to increase code coverage > - [x] Actually fixing the lexer > * In cases where is is **broken** > Or it does **Not yet** implement everything stipulated in the grammar > * Finding what the lexer does but grammar does not yet document and updating the docs accordingly > - [x] Adding tab support as per #121 > - [x] Numerical encoders `21UL` etc. > * @deavmi and @gusmeyer should discuss this > * Entails usage of `isNumericalEncoder(char character)`, `isNumericalEncoder_Size(char character)` and `isNumericalEncoder_Signage(char character)` > - [x] Implement `isValidEscape_String(char)` > * Might need more `\\` things etc required > * We should make a list of supported excape sequences we want, add support for them and so on > - [x] String support > - [x] Character support (like `'\n'`) > - [x] Enum for character constants frequently used > - [x] Dotted handling edge cases `new A()\t.\tl . p.p; ` > - [x] Bug: The lexer currently allows spaces/tabs around a standalone '.' or with spaces on either side, which should no tbe the case. > - [x] Investigate #60 > - [x] Support for comments #84 > - [x] Support for `2_100` > * This should become a token `[2100]` > - [x] Float valid-splitters function > * Determine valid splitters for float > - [ ] Once done speak to @deavmi as you need to document the code as well as in the documentation/manual > - [x] Move `isSplitter(char)`, `isOperator(char)` to `core/lexer.d` @deavmi > - [x] Moved > - [x] Old ones to use migrated ones Migrated all methods that were common `char`-based ones
deavmi stopped working 2023-12-24 13:41:17 +00:00
3 minutes 13 seconds
deavmi started working 2023-12-24 14:29:18 +00:00
deavmi stopped working 2023-12-24 14:29:23 +00:00
5 seconds
deavmi added spent time 2023-12-24 14:29:28 +00:00
10 minutes
deavmi started working 2023-12-24 14:29:31 +00:00
deavmi commented 2023-12-24 14:30:51 +00:00
Author
Owner
Copy Link

Looks like I have everything done regarding documentation and neatening up of source code.

Ready to merge ✅

Looks like I have everything done regarding documentation and neatening up of source code. Ready to merge ✅
deavmi stopped working 2023-12-24 14:30:56 +00:00
1 minute 25 seconds
deavmi started working 2023-12-24 14:57:39 +00:00
deavmi stopped working 2023-12-24 14:59:10 +00:00
1 minute 31 seconds
deavmi started working 2023-12-27 06:12:34 +00:00
deavmi commented 2023-12-27 06:12:46 +00:00
Author
Owner
Copy Link

Just looking for any undocumented methods right now

Just looking for any undocumented methods right now
deavmi commented 2023-12-27 06:16:33 +00:00
Author
Owner
Copy Link

Just looking for any undocumented methods right now

Done ✅

> Just looking for any undocumented methods right now Done ✅
deavmi stopped working 2023-12-27 06:16:38 +00:00
4 minutes 5 seconds
deavmi started working 2023-12-27 06:17:48 +00:00
deavmi commented 2023-12-27 06:18:08 +00:00
Author
Owner
Copy Link

Merging...

Merging...
deavmi commented 2023-12-27 06:18:27 +00:00
Author
Owner
Copy Link

Merged ✅

Merged ✅
deavmi commented 2023-12-27 06:18:50 +00:00
Author
Owner
Copy Link

CI/CD running...

CI/CD running...
deavmi commented 2023-12-27 06:19:27 +00:00
Author
Owner
Copy Link

Unittests ✅ (local testing)

Unittests ✅ (local testing)
deavmi commented 2023-12-27 06:19:49 +00:00
Author
Owner
Copy Link

CI/CD ✅

CI/CD ✅
deavmi commented 2023-12-27 06:19:58 +00:00
Author
Owner
Copy Link

Done, closing issue ✅

Done, closing issue ✅
deavmi stopped working 2023-12-27 06:20:08 +00:00
2 minutes 20 seconds
Sign in to join this conversation.
feature/lexer_improvements
Branches Tags
Clear current reference
master
feature/aliases
vardec_varass_dependency
cleanup/depgen_root_removal
feature/modifiers
bugfix/dgen_expressionless_returns
cleanup/remove_dnodegeneratorbase
feature/comments
feature/operator_precedence
feature/reporting
bugfix/single_line_comment_bug
feature/logging_upgrade
feature/multi_module
feature/unused_vars_detection
feature/container_methods
feature/strat_resolve_collectors
feature/predicate_based_resolver
feature/poolmngr
hotfix/func_defs_in_dep_generator
feature/lexer_improvements
feature/structs
feature/behavec_preinline
bugfix/fix_multi_compile
feature/annotations
hotfix/simply_varexp_dnode_generation
hotfix/jcli_upgrade_fix
feature/class_static_init
feature/universal_coercion
feature/cloneability
hotfix/expect_exception_change
bugfix/recursive_return_statement_check
bugfix/require_return
bugfix/return_type
feature/cleanup1
feature/clean_lexer_interface
research/gmalloc
feature/type_enforcer_cast_instr_emit
feature/meta_programming
feature/direct_func_call
bugfix/return_position
bugfix/is_same_type_fallback
feature/lexer_interface
feature/parser_interface
arrays
pointers
bugfix/function_parameter_type_parse
casting
compiler_object
extern_symbols
literal_encodings
uniform_stdalone_varass
remove_typequeue
linearizer_cleanup
parsetypedev_fix
modules
loops
conditionals
vdva_func_param_merge
function_parameters_fix
path_dependency
function_dep
compiler_levels
parser
constants
constants_float
develop
typecheck
deps_refactor
codegen
lexer
typecheck_refactor_varexp_makeacyclic
weighting
testing_thing
typecheck_refactor_varexp
typecheck_refactor
type_check_codegen_reducedparseclass_excep
typecheck_codegen
typecheck_codegen_educedparseclass
typecheck_reliancetree
static_support
codegen.sync-conflict-20210606-150925-O3W7KWN
develop.sync-conflict-20210606-150924-O3W7KWN
struct_support
typecheck.sync-conflict-20210606-150925-O3W7KWN
entity_declaration_type_checking
symbol_table
parser.sync-conflict-20210405-185821-O3W7KWN
parser_exception_before
develop_before_lexer_parser_merge_parseName
if_statement_parsing
exception_except
expression_parsing
master.sync-conflict-20210316-165421-O3W7KWN
expression_parsing.sync-conflict-20210316-090018-O3W7KWN
Clear current reference
multimodule/changeset2
multimodule/changeset1
pointer_cast_merge
parsetypeddeclaration_onlytype_parsecast_new
pointers_coerce
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.4.0
v0.3.0
v0.2.0
v0.1.0
v0.0.5
v0.0.4
v0.0.3
v0.0.2
v0.0.1
Labels
Clear labels   
dependency

Dependency tree generation

  
emit

Code emitting

  
hashmapper

Anything pertaining to the HashMapper symbol mapping code

  
lebanonmapper

Anything pertaining to the LebanonMapper symbol mapping code

  
lexer

Related to the tokenizer

  
meta

Meta processor related

  
needsfix

Urgent fix required

  
parser

Parser related

  
qol

Quality-of-life related

  
question

A question, not needed fix

  
resolution

Related to the resolver

  
typing

TypeChecker and Resolver related

No Label
dependency
emit
hashmapper
lebanonmapper
lexer
meta
needsfix
parser
qol
question
resolution
typing
Milestone
Clear milestone
No items
No Milestone
Basics
Projects
Clear projects
No project
Lexer
Assignees
Clear assignees
No Assignees
gusmeyer
deavmi
1 Participants
Notifications
Total Time Spent: 4 hours 29 minutes
deavmi
2 hours 7 minutes
gusmeyer
2 hours 21 minutes
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

2023-12-31

Blocks
#157 ⚡ Feature: Multi-module support
tlang/tlang
Depends on
#121 🐞️ Bugfix: Tabs in lexer: Not handled
tlang/tlang
You do not have permission to read 2 dependencies
Reference: tlang/tlang#144
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?