|
Posted by Ton 't Lam on November 2, 2005, 3:15 pm
Please log in for more thread options
A) I did:
# cd ~/.cpan/build/Parse-Flex-0.03
# more src/default.y
%{
#define YY_DECL char* yyylex YY_PROTO(( void ))
#undef yywrap
int yywrap(void) { return 1 ;}
char buffer[15];
%}
%%
[a-z]+ { return strcpy(buffer, "VAR"); }
[0-9]+ { return strcpy(buffer, "NUM"); }
[ \t]+ ;
.. { return strcpy(buffer, yytext); }
\n { return strcpy(buffer, yytext); }
<> return "" ;
%%
And then make and install. A quick test
$ perl -MParse::Flex -e 'print join " ", yylex(),"\n"'
abc
VAR abc
I borrowed the Calc example from Parse::Yapp, but want to use Flex
instead, thus
# cat Calc
use Calc;
use Parse::Flex;
sub yapp_error {
my $self = shift;
warn "Error: found ", $self->YYCurtok,
" and expecting one of ", join(" or ",$self->YYExpect);
}
my $calc = new Calc;
$calc->YYParse( yylex => yylex, yyerror => \&yapp_error );
This however results in:
Invalid value for parameter 'YYLEX' at Calculator line 12
I think the reason is that the yylex is also invoking the Parse::Flex
the first time.
I changed Parse::Flex, and renames yylex in flex:
$calc->YYParse( yylex => flex, yyerror => \&yapp_error );
to no avail. I can do
($token,$value) = flex;
What do I miss/oversee here ??
B) For sure I wrote my own My::FLex, with a different approach. In a
nutshell (I'm happy to append details on request)
In Flex.pm:
sub flex {
my $token = yylex();
if ($token) {
my $value = yytext();
# print $token, " = ", $value, "\n";
return ($token, $value);
}
else {
# print "Undefined token\n";
return (undef, "");
}
}
yylex()
OUTPUT:
RETVAL
char*
yytext()
CODE:
RETVAL = yytext;
OUTPUT:
RETVAL
Make and install the module and simply working.
my $calc = new Calc;
$calc->YYParse( yylex => flex, yyerror => \&yapp_error );
and that is the same invokation as above.
Best Regards,
Ton 't Lam
|
|
Posted by Ton 't Lam on November 3, 2005, 11:50 am
Please log in for more thread options
General
Parse::Yapp and flex is a powerful combination to do grammar and pattern
matching. You have the joy of the fast lexical analyzer, and within Perl
you create a program easily.
Yapp expects two variables, and per consequence the flex module should
provide these. The problem is that both Parse::Yapp, and flex use yylex.
I couldn't find Perl syntax to force the first yylex to be from Parse::Yapp.
$calc->YYParse( yylex => yylex, ...);
To go short: I changed Parse::Flex, as well created My::Flex. The fatest
method is to use Parse::Flex.
Parse::Flex
About the man page
The Parse::Flex manual says:
yylex() Get the name of next token, and indirectly sets yytext . Returns
undef for end of input.
You should read this as:
yylex() Get the name of next token, and indirectly sets yytext . Returns
undef for end of input.
Two variables are returned.
yying = 'datafile'
should be:
yyin = 'datafile'
Modifying Parse::Flex
Modify Flex.xs:
void
flex()
PPCODE:
char* id = 0;
if (id = yyylex() ) {
XPUSHs (sv_2mortal(newSVpv(id,0)));
XPUSHs (sv_2mortal(newSVpv( yytext, 0)));
XSRETURN(2);
}
XSRETURN_EMPTY;
Modify Flex.pm
our @EXPORT = qw( yyin yyout flex );
and src/default.y contains e.g.:
%{
#define YY_DECL char* yyylex YY_PROTO(( void ))
#undef yywrap
int yywrap(void) { return 1 ;}
%}
%%
[a-z]+ { return "VAR"; }
[0-9]+ { return "NUM"; }
[ \t]+ ;
.. ECHO;
\n ECHO;
<> return "" ;
%%
# make
# make install
A quick test
$ perl -MParse::Flex -e 'print join " ", flex(),"\n"'
abc
VAR abc
Yapp
Install Parse::Yapp
# cpan -i Parse::Yapp
Take the Calc.yp example from Parse::Yapp. We will use this example, but
of course use the flex lexer instead. Parse::Yapp expects a yylex
function that returns a pair: the name of the token and the matched text.
From the Calc.yp keep
#
# Calc.yp
#
# Parse::Yapp input grammar example.
#
# This file is PUBLIC DOMAIN
#
#
%right '='
%left '-' '+'
%left '*' '/'
%left NEG
%right '^'
%%
input: #empty
| input line { push(@,$_[2]); $_[1] }
;
line: '\n' { $_[1] }
| exp '\n' { print "$_[1]\n" }
| error '\n' { $_[0]->YYErrok }
;
exp: NUM
| VAR { $_[0]->YYData-> }
| VAR '=' exp { $_[0]->YYData->=$_[3] }
| exp '+' exp { $_[1] + $_[3] }
| exp '-' exp { $_[1] - $_[3] }
| exp '*' exp { $_[1] * $_[3] }
| exp '/' exp {
$_[3]
and return($_[1] / $_[3]);
$_[0]->YYData->
= "Illegal division by zero.\n";
$_[0]->YYError;
undef
}
| '-' exp %prec NEG { -$_[2] }
| exp '^' exp { $_[1] ** $_[3] }
| '(' exp ')' { $_[2] }
;
%%
sub yapp_error {
my $self = shift;
warn "Error: found ", $self->YYCurtok,
" and expecting one of ", join(" or ",$self->YYExpect);
}
# yapp Calc.yp
The result is a Calc.pm in your current directory.
Yapp invokes flex
# cat Calc
use Calc;
use Parse::Flex;
sub yapp_error {
my $self = shift;
warn "Error: found ", $self->YYCurtok,
" and expecting one of ", join(" or ",$self->YYExpect);
}
my $calc = new Calc;
# $calc->YYParse( yylex => \&My::Flex::flex, yyerror => \&Calc::yapp_error);
$calc->YYParse( yylex => \&flex, yyerror => \&Calc::yapp_error);
# perl Calc
12+34
46
12++34
Error: found + and expecting one of - or NUM or VAR or ( at Calc.yp line 48.
My::Flex
Here are steps to create your own flex module My::Flex
Here:
* Parse::Flex supplies flex (Flex.pm)
* flex (Flex.pm) calls yylex, and yytext (Flex.xs).
* Flex.xs calls the functions from lex.yy.o, thus Flex.l.
Next step is to build an XS module, that invokes flex to do the pattern
matching.
Create the module structure:
# h2xs -O -n My::Flex
# cd My-Flex
Create a file MyFlex.h
# more MyFlex.h
extern char* yylex();
extern char* yytext;
extern FILE *yyin, *yyout;
Modify Flex.xs, so that
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include "const-c.inc"
#include "myFlex.h"
MODULE = My::Flex PACKAGE = My::Flex
INCLUDE: const-xs.inc
char*
yylex()
CODE:
RETVAL = yylex();
OUTPUT:
RETVAL
char*
yytext()
CODE:
RETVAL = yytext;
OUTPUT:
RETVAL
void
yyin( file)
char* file
CODE:
yyin = fopen( file, "r");
void
yyout( file)
char* file
CODE:
yyout = fopen( file, "w");
and the myFlex.l
# more myFlex.l
%{
#define YY_DECL char* yylex() void;
char buffer[15];
%}
%%
apple ECHO;
pear ECHO;
[a-z]+ { return strcpy(buffer, "VAR"); }
[0-9]+ { return strcpy(buffer, "NUM"); }
[ \t]+ ;
\n { return strcpy(buffer, yytext); }
.. { return strcpy(buffer, yytext); }
%%
int perl_yywrap(void) {
return 1;
}
We have to map the data types from C to Perl. Next map translates the
pointers to characters to the built-in T_PV Perl type.
# cat typemap
char* T_PV
In lib/My/Flex.pm enter
# Preloaded methods go here.
sub flex {
my $token = yylex();
if ($token) {
my $value = yytext();
# print $token, " = ", $value, "\n";
return ($token, $value);
}
else {
# print "Undefined token\n";
return (undef, "");
}
}
Now the lib/My/Flex.pm
# our %EXPORT_TAGS = ( 'all' => [ qw( ) ] );
# our @EXPORT_OK = ( @{ $EXPORT_TAGS } );
our @EXPORT = qw(yyin yyout flex );
....
sub flex {
my $token = yylex();
if ($token) {
my $value = yytext();
# print $token, " = ", $value, "\n";
return ($token, $value);
}
else {
# print "Undefined token\n";
return (undef, "");
}
}
Next step is to modify the Makefile.PL
WriteMakefile(
NAME => 'My::Flex',
VERSION_FROM => 'lib/My/Flex.pm', # finds $VERSION
PREREQ_PM => {}, # e.g., Module::Name => 1.1
($] >= 5.005 ? ## Add these new keywords supported since 5.005
(ABSTRACT_FROM => 'lib/My/Flex.pm', # retrieve abstract from module
AUTHOR => 'Ton ') : ()),
LIBS => ['-lfl'], # e.g., '-lm'
DEFINE => '', # e.g., '-DHAVE_SOMETHING'
INC => '-I.', # e.g., '-I. -I/usr/include/other'
MYEXTLIB => 'myFlex.so' # our Flexer (note there is no comma)
# Un-comment this if you add C files to link with later:
# OBJECT => '$(O_FILES)', # link all the C files too
);
<snip>
sub MY::postamble {
"
$(MYEXTLIB): lex.yy.c Flex.o
\t$(CC) -c lex.yy.c
\t$(AR) cr $(MYEXTLIB) lex.yy.o
\tranlib $(MYEXTLIB)
lex.yy.c: myFlex.l
\tflex myFlex.l
";
}
It is time to compile
# perl Makefile.PL
# make
# make test # FWIW
# make install
If you don have root access, then do
$ perl Makefile.PL PREFIX=/home/tonl
$ make
$ make install
If you want to re-compile then remove the *.[co] files first, or make clean.
Test if the module is installed.
# perl -MMy::Flex -e 1
A quick test
# perl -MMy::Flex -e 'print join " ", My::Flex::flex(),"\n"'
abc
VAR abc
And finally invoke the modules, and do the calculation ...
# use lib "/home/tonl"; # if you installed locally
use Calc;
use My::Flex;
My::Flex::yyin("file");
My::Flex::yyout "res";
My::Flex::flex();
print "===\n";
my $calc = new Calc;
$calc->YYParse( yylex => \&flex, yyerror => \&Calc::yapp_error);
and file contains
apple
pear
1+2
abc=4
def=5
abc+def
apple
# perl Calc
===
3
4
5
9
and res contains
applepearapple
Note about yyout:
flex its ECHO directs output to yyout, and the other output is return to
the Perl program.
I could have done eqally the following:
void
flex()
PPCODE:
char* id = 0;
if (id = yylex() ) {
XPUSHs (sv_2mortal(newSVpv(id,0)));
XPUSHs (sv_2mortal(newSVpv( yytext, 0)));
XSRETURN(2);
}
XSRETURN_EMPTY;
This will cause an Perl array to be returned. Then you don't need to
modify lib/My/Flex.pm.
$ perl -MMy::Flex -e 'print join " ", My::Flex::flex(),"\n"'
abc
VAR abc
use Calc;
use My::Flex;
my $calc = new Calc;
$calc->YYParse( yylex => \&My::Flex::flex(), yyerror => \&Calc::yapp_error);
References
* Parse::Yapp
* Parse::Flex
* perlxstut
Author
Ton 't Lam
|
|
Posted by Ton 't Lam on November 4, 2005, 6:46 pm
Please log in for more thread options
You may come accross next error message:
Usage: Parse::Flex::flex() at
/opt/perl/lib/site_perl/5.8.2/Parse/Yapp/Driver.pm line 271.
and if you try that
Invalid value for parameter 'YYLEX' at Calc line 11
The reason is that in Driver.pm next invokation is made:
($$token,$$value)=&$lex($self);
In other words, a parameter is expected. flex() doesn't accept this. The
workaround is to have next:
use Calc;
use Parse::Flex;
sub _Flex {
flex; # or Parse::Flex::flex;
}
my $calc = new Calc;
$calc->YYParse( yylex => \&_Flex, yyerror => \&Calc::yapp_error);
Another easy solution is to have flex accept the variable.
void
flex(path)
SV * path;
PPCODE:
and then you can have:
my $calc = new Calc;
$calc->YYParse( yylex => \&flex, yyerror => \&Calc::yapp_error );
Remark: This happened with Parse::Flex, but not with My::Flex.
Linefeed not accepted
Unknown why this is. Entering a second linefeed is accepted.
# echo "1+2" | perl Calc
works as expected. Also if using
yyin "file";
Remark: This happened with Parse::Flex, but not with My::Flex.
|
| Similar Threads | Posted | | Extending Parse::Yapp with new functionalities | February 6, 2007, 7:29 am |
| Problem to install Parse::Lex, all tests fails | October 24, 2005, 8:15 pm |
| Parse::Readelf 0.01 - parse the output of readelf | September 24, 2007, 12:07 pm |
| Problems with Parse::Lex | May 5, 2005, 8:22 pm |
| I want to take over module Parse::Lex, how to do it? | April 17, 2006, 10:49 pm |
| Parse::FixedLength. where did pars() go? | September 30, 2004, 2:02 pm |
| How to parse with regular expression... | May 5, 2005, 6:33 am |
| Parse::RecDescent demo_Cgrammar.pl | March 14, 2006, 6:17 pm |
| Any modules to parse ELF files? | August 7, 2007, 5:09 pm |
| three Parse::RecDescent related questions | October 7, 2005, 1:23 pm |
|
XML Sitemap