|
Posted by iaminsik on May 6, 2008, 4:00 am
Please log in for more thread options
In most cases, I converted utf-16le files into utf-8 encoding.
But, I want to handle utf-16le files directly.
My first source is "read a line from utf-16le file and write it in
utf-16le encoding".
It works well.
==========================================================
use utf8;
use Encode;
open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
binmode $infile;
open ($outfile, ">:raw:encoding(UTF-16LE):crlf", "unicodefile.out");
binmode $outfile;
while ($line = <$infile>)
{
print $outfile $line;
}
close($infile);
close($outfile);
==========================================================
Second source is "read one line, split it into array, and print array
by line in utf-16le encoding".
It seemed to work well, but some characters were broken. It didn't
work well.
After a long web searching, I recognized Unicode::String could solve
this problem.
==========================================================
use utf8;
use Encode;
$\ = "\n";
open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
binmode $infile;
open ($outfile, ">:raw:encoding(UTF-16LE):crlf", "unicodefile.out");
binmode $outfile;
while ($line = <$infile>)
{
chomp($line);
@words = split(/[ ]+/, $line);
foreach $word (@words)
{
print $outfile $word;
}
}
close($infile);
close($outfile);
==========================================================
Using Unicode::String, I made the third source, but still it doesn't
work.
It means "reading" is OK, but split function isn't.
Is there any solution?
==========================================================
use utf8;
use Encode;
use Unicode::String;
Unicode::String->stringify_as('utf16');
$\ = "\n";
open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
binmode $infile;
open ($outfile, ">:raw:encoding(UTF-16LE):crlf", "unicodefile.out");
binmode $outfile;
while ($line = <$infile>)
{
chomp($line);
$sep = new Unicode::String ("[ ]+");
@words = split($sep, $line);
foreach $word (@words)
{
print $outfile $word;
}
}
close($infile);
close($outfile);
==========================================================
Best Regards.
Remi
|
|
Posted by Ben Bullock on May 6, 2008, 6:44 am
Please log in for more thread options
On Tue, 06 May 2008 01:00:50 -0700, iaminsik wrote:
> In most cases, I converted utf-16le files into utf-8 encoding. But, I
> want to handle utf-16le files directly.
>
> My first source is "read a line from utf-16le file and write it in
> utf-16le encoding".
> It works well.
No it doesn't. Your problems are all in the first file.
> open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
> binmode $infile;
> open ($outfile, ">:raw:encoding(UTF-16LE):crlf", "unicodefile.out");
> binmode $outfile;
Do you know what binmode does? You'd better have another look at the
manual (perldoc -f binmode). The binmode statements here switch OFF all
the :raw:encoding(UTF... stuff you'd put in the previous lines, which
explains all the other problems you had.
To demonstrate, try this:
#!/usr/local/bin/perl
use warnings;
use strict;
use utf8;
use Encode;
binmode STDOUT, "utf8";
my $utf8 = "モンスター 自惚れ";
for (qw/file1 file2/) {
open (my $outfile, ">:raw:encoding(UTF-16LE):crlf", "$_.dat") or die
$!;
binmode $outfile if /1/; # do what you did for file1 only
print $outfile $utf8;
close $outfile or die $!;
open (my $infile, "<:encoding(UTF-16LE):crlf", "$_.dat");
while (my $line = <$infile>)
{
print "$_: $line\n";
}
close($infile) or die $!;
}
The reason your code appeared to work is because you never did anything
with the data. It was actually just reading and writing it as bytes
without any knowledge of the encoding. As soon as you tried to manipulate
the data, the problem which had been there all along became visible.
P.S. use warnings; use strict; & check the values of open and close as
above.
|
|
Posted by Ben Bullock on May 6, 2008, 6:57 am
Please log in for more thread options On Tue, 06 May 2008 10:44:09 +0000, Ben Bullock wrote:
> open (my $infile, "<:encoding(UTF-16LE):crlf", "$_.dat");
> P.S. use warnings; use strict; & check the values of open and close as
> above.
Oops!
|
|
Posted by Ben Bullock on May 7, 2008, 1:46 am
Please log in for more thread options
> The first source generates 'wide character warnings',
> and saves outfile in utf8 format, weirdly.
It's not weird; you have "use utf8;" there, so it reads in using the
encoding you specified, then the binmode switches off the output
formatting, then it prints it out in the default format, which
generates wide character warnings because you haven't explicitly set
the mode of the output to anything. Use
binmode $outfile,"utf8";
to switch those wide character warnings off.
> I made 'binmode $outfile;' as a comment line,
> and it saves outfile in UTF-16LE format I wanted.
Good news.
> 3. Several Questions
> I, a newbie in Perl programming language, couldn't understand two
> parts in your codes.
> ========================================================================
> for (qw/file1 file2/) { <===== what it means? it's a short expression
> for loop?
This sets $_ to "file1" then "file2". qw/a b/ equals ('a', 'b').
> binmode $outfile if /1/; <===== what /l/ means?
It's not an l it's a 1. "if /1/" has the effect of saying 'if $_ is
"file1"'. The /1/ detects the character '1' in the name. Try
experimenting with the code to understand what it does.
|
|
Posted by iaminsik on May 7, 2008, 11:38 pm
Please log in for more thread options On 5=BF=F97=C0=CF, =BF=C0=C8=C42=BD=C346=BA=D0, benkasminbull...@gmail.com (=
Ben Bullock) wrote:
> > The first source generates 'wide character warnings',
> > and saves outfile in utf8 format, weirdly.
>
> It's not weird; you have "use utf8;" there, so it reads in using the
> encoding you specified, then the binmode switches off the output
> formatting, then it prints it out in the default format, which
> generates wide character warnings because you haven't explicitly set
> the mode of the output to anything. Use
>
> binmode $outfile,"utf8";
>
> to switch those wide character warnings off.
>
> > I made 'binmode $outfile;' as a comment line,
> > and it saves outfile in UTF-16LE format I wanted.
>
> Good news.
>
> > 3. Several Questions
> > I, a newbie in Perl programming language, couldn't understand two
> > parts in your codes.
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > for (qw/file1 file2/) { <=3D=3D=3D=3D=3D what it means? it's a short exp=
ression
> > for loop?
>
> This sets $_ to "file1" then "file2". qw/a b/ equals ('a', 'b').
>
> > binmode $outfile if /1/; <=3D=3D=3D=3D=3D what /l/ means?
>
> It's not an l it's a 1. "if /1/" has the effect of saying 'if $_ is
> "file1"'. The /1/ detects the character '1' in the name. Try
> experimenting with the code to understand what it does.
Your comment helped me a lot.
Thanks, Ben!
Best Regards,
Remi.
|
| Similar Threads | Posted | | Problem handling a Unicode file | August 28, 2006, 2:34 am |
| cp error handling | September 17, 2004, 4:06 am |
| Error handling | July 4, 2007, 6:21 am |
| Error: unicode character property definition | March 26, 2006, 10:32 am |
| Net::Google error handling | March 7, 2005, 9:02 am |
| Error Handling in Net::SSH::Perl | November 13, 2006, 3:58 pm |
| Best error handling mechanism? | June 4, 2007, 9:13 pm |
| mySQL error handling with PERL | June 21, 2005, 9:37 am |
| Net::Telnet Error handling/trapping | February 5, 2007, 4:38 pm |
| unicode string | March 1, 2006, 5:27 am |
|
XML Sitemap