add ability to limit number of expands

Created by: jlongster

This is a first pass at telling the expander to only perform n expansions on a file, which lets you walk through the expansion process which is really helpful for debugging (and understanding sweet.js in general).

I don't like abusing global variables, but having to constantly pass around the level of expansion would be crazy too. What do you think of this approach? This seems to work well except for a caveat that I need your help with, which I'll get to in a second.

This forced me to improve the output of the non-parsed source, too (the -p option), because limiting expansion essentially forces you to use it. I implemented a naive way to output the combined tokens in a nested format instead one long string, which quickly got unreadable. This naive nested form approximates the structure and helps a lot when scanning through it. It just looks for { and } and indents code when a new block is found.

Here's my example which motivated me to do this: https://gist.github.com/jlongster/7923415. Macros generate other macros multiple times, so it's really helpful to see the actual macro the is generated at each step.

There's a problem with my current implementation: when a macro generates a macro, for some reason the resulting macro is lost from the source. The expansion-stepping works fine, but I don't see the intermediate macros. What am I doing wrong?

Here's what stepping looks like with my type macro from the gist:

Stepping once generates macro foo { ... } which creates a type. But the macro is lost:

% ./bin/sjs -p -n 1  ../stack-macro/src/macros.js
var MB = 1024 * 1024;
var STACK_SIZE = 2 * MB;
var SP = STACK_SIZE;
var buffer = new ArrayBuffer ( STACK_SIZE );
var U1 = new Uint8Array ( buffer );
var I1 = new Int8Array ( buffer );
var U2 = new Uint16Array ( buffer );
var I2 = new Int16Array ( buffer );
var U4 = new Uint32Array ( buffer );
var I4 = new Int32Array ( buffer );
var F4 = new Float32Array ( buffer );
var F8 = new Float64Array ( buffer );
Point foo;
foo . x;
foo . y;
Point bar;
bar . x;
bar . y;

You can see that this is working though when I allow 2 expansion steps:

% ./bin/sjs -p -n 2  ../stack-macro/src/macros.js
var MB = 1024 * 1024;                                     
var STACK_SIZE = 2 * MB;
var SP = STACK_SIZE;
var buffer = new ArrayBuffer ( STACK_SIZE );
var U1 = new Uint8Array ( buffer );
var I1 = new Int8Array ( buffer );
var U2 = new Uint16Array ( buffer );
var I2 = new Int16Array ( buffer );
var U4 = new Uint32Array ( buffer );
var I4 = new Int32Array ( buffer );
var F4 = new Float32Array ( buffer );
var F8 = new Float64Array ( buffer );
var ptr = SP;
SP -= 16;
;
foo . x;
foo . y;
Point bar;
bar . x;
bar . y;

And 3:

% ./bin/sjs -p -n 3 ../stack-macro/src/macros.js
var MB = 1024 * 1024;
var STACK_SIZE = 2 * MB;
var SP = STACK_SIZE;
var buffer = new ArrayBuffer ( STACK_SIZE );
var U1 = new Uint8Array ( buffer );
var I1 = new Int8Array ( buffer );
var U2 = new Uint16Array ( buffer );
var I2 = new Int16Array ( buffer );
var U4 = new Uint32Array ( buffer );
var I4 = new Int32Array ( buffer );
var F4 = new Float32Array ( buffer );
var F8 = new Float64Array ( buffer );
var ptr = SP;
SP -= 16;
;
lookup_view double [ ptr + 0 ];
foo . y;
Point bar;
bar . x;
bar . y;

And 4, 5, 6, and 7:

jlong-16106:~/projects/sweet.js(master)% ./bin/sjs -p -n 4 ../stack-macro/src/macros.js
var MB = 1024 * 1024;
var STACK_SIZE = 2 * MB;
var SP = STACK_SIZE;
var buffer = new ArrayBuffer ( STACK_SIZE );
var U1 = new Uint8Array ( buffer );
var I1 = new Int8Array ( buffer );
var U2 = new Uint16Array ( buffer );
var I2 = new Int16Array ( buffer );
var U4 = new Uint32Array ( buffer );
var I4 = new Int32Array ( buffer );
var F4 = new Float32Array ( buffer );
var F8 = new Float64Array ( buffer );
var ptr = SP;
SP -= 16;
;
F8 [ ptr + 0 ];
foo . y;
Point bar;
bar . x;
bar . y;

jlong-16106:~/projects/sweet.js(master)% ./bin/sjs -p -n 5 ../stack-macro/src/macros.js
var MB = 1024 * 1024;
var STACK_SIZE = 2 * MB;
var SP = STACK_SIZE;
var buffer = new ArrayBuffer ( STACK_SIZE );
var U1 = new Uint8Array ( buffer );
var I1 = new Int8Array ( buffer );
var U2 = new Uint16Array ( buffer );
var I2 = new Int16Array ( buffer );
var U4 = new Uint32Array ( buffer );
var I4 = new Int32Array ( buffer );
var F4 = new Float32Array ( buffer );
var F8 = new Float64Array ( buffer );
var ptr = SP;
SP -= 16;
;
F8 [ ptr + 0 ];
lookup_view double [ ptr + 8 ];
Point bar;
bar . x;
bar . y;

jlong-16106:~/projects/sweet.js(master)% ./bin/sjs -p -n 6 ../stack-macro/src/macros.js
var MB = 1024 * 1024;
var STACK_SIZE = 2 * MB;
var SP = STACK_SIZE;
var buffer = new ArrayBuffer ( STACK_SIZE );
var U1 = new Uint8Array ( buffer );
var I1 = new Int8Array ( buffer );
var U2 = new Uint16Array ( buffer );
var I2 = new Int16Array ( buffer );
var U4 = new Uint32Array ( buffer );
var I4 = new Int32Array ( buffer );
var F4 = new Float32Array ( buffer );
var F8 = new Float64Array ( buffer );
var ptr = SP;
SP -= 16;
;
F8 [ ptr + 0 ];
F8 [ ptr + 8 ];
Point bar;
bar . x;
bar . y;

jlong-16106:~/projects/sweet.js(master)% ./bin/sjs -p -n 7 ../stack-macro/src/macros.js
var MB = 1024 * 1024;
var STACK_SIZE = 2 * MB;
var SP = STACK_SIZE;
var buffer = new ArrayBuffer ( STACK_SIZE );
var U1 = new Uint8Array ( buffer );
var I1 = new Int8Array ( buffer );
var U2 = new Uint16Array ( buffer );
var I2 = new Int16Array ( buffer );
var U4 = new Uint32Array ( buffer );
var I4 = new Int32Array ( buffer );
var F4 = new Float32Array ( buffer );
var F8 = new Float64Array ( buffer );
var ptr = SP;
SP -= 16;
;
F8 [ ptr + 0 ];
F8 [ ptr + 8 ];
var ptr = SP;
SP -= 16;
;
bar . x;
bar . y;

How can I improve this? Can you explain what the enforest stage does? I'm not entirely sure when to let it recurse into the step function and how to reason about that. I played around with it a bunch but still unclear how macro-creating macros play with the enforest and expandToTermTree stages.