Compiler From Scratch: Phase 1 - Tokenizer Generator 015: Finishing Lazy Token Evaluation
Streamed on 2024-10-25 (https://www.twitch.tv/thediscouragerofhesitancy)
Zero Dependencies Programming!
Last week we started with the Lazy evaluation build option, but didn't have time to finish it. So today we finished it.
As I dug in, it became clear that I needed a way to track execution of the regex DFA states to see what was happening and when. So I started coding that in. However, that introduced a bug that caused things to crash so hard that there was no debug information. After a log to printf debugging, I narrowed it down to a single snippet of text that was being written out to a file. For some reason, copying a single word out by offset using snprintf was exploding if the source text had a "%" anywhere in it . The "%" was not in the offset/count region that was actually being copied from at all. I replaced the call to snprintf with strncpy and everything worked again. It just makes me sad how much time it took to find the reason and the solution.
Once we were back up and running there was a bit of fiddling to finish off Lazy token evaluation. There are a few differences in the way the process flows when batch processing vs lazy processing, but I found them one at a time (greatly aided by the trace log of the tokenizing process). A bit of testing with other performance flags, an update to the macros in the resulting file, and lazy processing is reasonably working for all of the build options we tested.
Next week we need to write a script to test the different combinations more automatically and efficiently. There are a lot of build configuration options now.
-
1:02:46
FULL SEND PODCAST
22 hours agoNelk Boys REVEAL What Happened on Trump Force One and Talk Upcoming Trip to an Uncontacted Tribe!
30.9K19 -
12:49
Bearing
19 hours agoTikTok NIGHTMARE! It's Gotten Worse!
1.41K20 -
9:09
Degenerate Jay
13 hours ago $0.04 earnedBatman Is The First To Win This Huge Award!
2463 -
1:16:17
SonnyFaz
11 hours agoWe Need To Free America...
1.07K2 -
2:00:10
Danny Haiphong
13 hours ago $3.11 earnedBREAKING: Israel Launches ATTACK on Iran, Putin Warns of WW3 as BRICS Crushes Dollar w/ Ben Norton
4.3K16 -
7:32
America First Policy Institute
15 hours agoAFPI Weekly Rundown | Border Security, Bidenomics, & Integrity of Women’s Sports #news
13 -
4:01:54
The Original Next Level Gaming
17 hours agoNLG's Friday Night w/Peter & Mike: Call of Duty Black Ops 6! IT HAS BEGUN!!!
65.4K13 -
2:32:19
Fresh and Fit
10 hours agoAfter Hours w/ Tommy Sotomayor & Dom Lucre
90.5K82 -
4:40
SeasonofMist
2 months agoGAEREA - Hope Shatters (Official Music Video)
32 -
2:04:49
CocktailsConsoles
11 hours agoLAST CHAPTER! | Murder By Numbers | Cocktails & Consoles Livestream
43.3K6