tlang/tlang
tlang
/
tlang
mirror of https://github.com/tbklang/tlang.git
2
0
Fork 0
Code Issues 40 Projects 5 Releases Wiki Activity

Streaming lexer #106

New Issue
Open
opened 2023-03-08 14:28:49 +00:00 by deavmi · 3 comments
deavmi commented 2023-03-08 14:28:49 +00:00
Owner
Copy Link

What is this?

This issue pertains to the feature addition that requires @rany implement a new lexer that works on the source code that is read in (one shot, we could discuss updating the interface provided by LexerInterface in #105 to read the source bit-by-bit) and lexers the next token only when nextToken() is called.

You will need to cache stuff as previousToken() is a thing.

# What is this? This issue pertains to the feature addition that requires @rany implement a new lexer that works on the source code that is read in (one shot, we could discuss updating the interface provided by `LexerInterface` in #105 to read the source bit-by-bit) and lexers the next token only when `nextToken()` is called. You will need to cache stuff as `previousToken()` is a thing.
Ghost was assigned by deavmi 2023-03-08 14:28:56 +00:00
deavmi added a new dependency 2023-03-08 14:29:17 +00:00
#105 Lexer interface
deavmi added a new dependency 2023-03-08 14:29:21 +00:00
#104 Lexer re-work
deavmi commented 2023-03-08 14:30:28 +00:00
Author
Owner
Copy Link

🔐️ Locking for now till I stabilize stuff and onboard @rany

🔐️ Locking for now till I stabilize stuff and onboard @rany
deavmi locked and limited conversation to collaborators 2023-03-08 14:30:44 +00:00
deavmi added this to the Clean ups milestone 2023-03-08 14:32:21 +00:00
deavmi added the
qol
lexer
 labels 2023-03-08 14:32:26 +00:00
deavmi added this to the Lexer project 2023-03-08 14:32:36 +00:00
deavmi removed this from the Lexer project 2023-03-08 14:32:42 +00:00
deavmi added this to the Parser project 2023-03-08 14:32:48 +00:00
deavmi added reference feature/lexer_interface 2023-04-20 11:05:27 +01:00
deavmi removed a dependency 2023-04-20 13:53:25 +01:00
#104 Lexer re-work
deavmi commented 2023-07-08 15:26:44 +01:00
Author
Owner
Copy Link

I should just do this, in near future.

I should just do this, in near future.
deavmi removed reference feature/lexer_interface 2023-07-09 11:42:05 +01:00
Ghost was unassigned by deavmi 2023-07-09 11:42:11 +01:00
deavmi commented 2023-07-09 11:42:26 +01:00
Author
Owner
Copy Link

This will be worked on later, not all too urgent for now at the very least.

This will be worked on later, not all too urgent for now at the very least.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
feature/modifiers
feature/comments
feature/operator_precedence
vardec_varass_dependency
feature/reporting
bugfix/single_line_comment_bug
feature/logging_upgrade
feature/multi_module
feature/unused_vars_detection
feature/container_methods
feature/strat_resolve_collectors
feature/predicate_based_resolver
feature/poolmngr
hotfix/func_defs_in_dep_generator
feature/lexer_improvements
feature/structs
feature/behavec_preinline
bugfix/fix_multi_compile
feature/annotations
hotfix/simply_varexp_dnode_generation
hotfix/jcli_upgrade_fix
feature/class_static_init
feature/universal_coercion
feature/cloneability
hotfix/expect_exception_change
bugfix/recursive_return_statement_check
bugfix/require_return
bugfix/return_type
feature/cleanup1
feature/clean_lexer_interface
research/gmalloc
feature/type_enforcer_cast_instr_emit
feature/meta_programming
feature/direct_func_call
bugfix/return_position
bugfix/is_same_type_fallback
feature/lexer_interface
feature/parser_interface
arrays
pointers
bugfix/function_parameter_type_parse
casting
compiler_object
extern_symbols
literal_encodings
uniform_stdalone_varass
remove_typequeue
linearizer_cleanup
parsetypedev_fix
modules
loops
conditionals
vdva_func_param_merge
function_parameters_fix
path_dependency
function_dep
compiler_levels
parser
constants
constants_float
develop
typecheck
deps_refactor
codegen
lexer
typecheck_refactor_varexp_makeacyclic
weighting
testing_thing
typecheck_refactor_varexp
typecheck_refactor
type_check_codegen_reducedparseclass_excep
typecheck_codegen
typecheck_codegen_educedparseclass
typecheck_reliancetree
static_support
codegen.sync-conflict-20210606-150925-O3W7KWN
develop.sync-conflict-20210606-150924-O3W7KWN
struct_support
typecheck.sync-conflict-20210606-150925-O3W7KWN
entity_declaration_type_checking
symbol_table
parser.sync-conflict-20210405-185821-O3W7KWN
parser_exception_before
develop_before_lexer_parser_merge_parseName
if_statement_parsing
exception_except
expression_parsing
master.sync-conflict-20210316-165421-O3W7KWN
expression_parsing.sync-conflict-20210316-090018-O3W7KWN
multimodule/changeset2
multimodule/changeset1
pointer_cast_merge
parsetypeddeclaration_onlytype_parsecast_new
pointers_coerce
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.4.0
v0.3.0
v0.2.0
v0.1.0
v0.0.5
v0.0.4
v0.0.3
v0.0.2
v0.0.1
Labels
Clear labels   
dependency

Dependency tree generation

  
emit

Code emitting

  
hashmapper

Anything pertaining to the HashMapper symbol mapping code

  
lebanonmapper

Anything pertaining to the LebanonMapper symbol mapping code

  
lexer

Related to the tokenizer

  
meta

Meta processor related

  
needsfix

Urgent fix required

  
parser

Parser related

  
qol

Quality-of-life related

  
question

A question, not needed fix

  
resolution

Related to the resolver

  
typing

TypeChecker and Resolver related

No Label
dependency
emit
hashmapper
lebanonmapper
lexer
meta
needsfix
parser
qol
question
resolution
typing
Milestone
Clear milestone
No items
No Milestone
Clean ups
Projects
Clear projects
No project
Parser
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Depends on
#105 Lexer interface
tlang/tlang
Reference: tlang/tlang#106
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?