Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • S sweet-core
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 62
    • Issues 62
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 4
    • Merge requests 4
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • sweet-js
  • sweet-core
  • Merge requests
  • !601

Implement tokenization via readtables

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Gabe Johnson requested to merge github/fork/gabejohnson/readtables into master Sep 14, 2016
  • Overview 35
  • Commits 18
  • Pipelines 0
  • Changes 31

Note: This work is definitely not complete, functional or ready to be merged.

This is my first (published) pass at a readtable implementation. It isn't hooked into the pipeline yet but does show the general strategy and initial API. Per some previous discussions https://github.com/sweet-js/sweet.js/issues/535#issuecomment-242791944 I've intentionally implemented a very limited API (utility libs can build on these).

Readtables can currently be created as follows:

import { defaultReadtable } from 'src/readtable';
import CharStream from 'src/char-stream';

const newTable = defaultReadtable.extendReadtable({
  key: '@',
  action(stream) {
    return {
      type: 'Punctuator',
      value: stream.readString(),
      locationInfo: stream.locationInfo
    };
  }
});

const entry = newTable.getEntry('@');
const stream = new CharStream('@foo');
entry.action(stream);
// =>
// {
//   type: 'Punctuator',
//   value: '@',
//   locationInfo: {
//     filename: '',
//     line: 0,
//     column: 0,
//     position: 0
//   }
// }

The intention is to allow lookbehind of already read tokens. I'll have to implement this for '/' resolution. So the signature of action will be action(stream: CharStream, prefix: TokenTree?). TokenTree = Token | List<TokenTree>.

I have not implemented Racket style reader modes as I believe they can be build using the current primitives (i.e. by referencing other readtables in a closure).

As a note, I'm using some "private" fields (denoted by a _ prefix). I'm reserving the right to truly hide them in the future if there is a reason and efficiencies don't suffer too greatly. So their presence shouldn't be relied upon and shouldn't be part of the public API.

@disnet I'd appreciate some feedback as soon as you get the chance. I'm going to have a lot of time the next few days to hack on this and don't want to go too far down the wrong path.

Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: github/fork/gabejohnson/readtables