C++ Tokenizer with Multiple Delimiters

Nvidia / Nvidia nforce 590 sli
July 21, 2009 at 13:18:47
Specs: Microsoft Windows XP Home Edition, 2.009 GHz / 2559 MB
For one of my classes, I'm making a compiler using C++. I got most of the things set, but I need to Tokenize my input file.

This isn't too hard in its own, but I have to be able to make tokens out of text like this:

int main()
{ var char a; int b; }

However, both "(" and ")" are tokens, as are "{" and "}". The way I see it, though, they are also delimiters.

So, I need help making a Tokenizer that finds the following as delimiters: " ", "{", "}", "(", ")" and also returns the delimiters so I can include them as tokens.

Searching around the internet, I haven't found any help with this exact case.

Any things to help me get started or suggestions would be appreciated.

See More: C++ Tokenizer with Multiple Delimiters

Report •

July 21, 2009 at 13:33:57
You probably won't, either. Having your delimiters included as separate tokens is kind of a specialized case. What do you have so far?

Report •

July 21, 2009 at 13:41:54
Well, the majority of my program involves other things at the moment. As far as code for the Tokenizing section, I have deleted and changed it so much now that it really isn't much.

I suppose I could just edit my input files to put spaces between everything, and take the easy route out...huh?

Report •

July 21, 2009 at 14:13:03
I suppose; it shouldn't be too hard to write your own, though. Especially if you use the string class and STL's list class.

Report •

Related Solutions

Ask Question