|
Posted by Ela on April 11, 2008, 8:46 am
Please log in for more thread options
except in using mysql or related programs, how can perl be used to deal with
large databases? Furthermore, how can I load that large db into cache in
advance and later use another perl to access/search that db?
|
|
Posted by Ted Zlatanov on April 11, 2008, 10:31 am
Please log in for more thread options
E> except in using mysql or related programs, how can perl be used to deal with
E> large databases?
In order (you may already know some of these):
1) learn SQL and at least some RDBMS design theory (select+join, update,
delete, indexing, foreign keys, normalization, data types, etc.)
2) read the DBI documentation
3) try Rose::DB::Object or Class::DBIx to see if they will work for you
instead of direct DBI queries.
I hope by "large" you mean at most a hundred gigabytes of data (which is
considered a mid-size database by most people). Large databases are
usually in the hundreds of gigabytes - terabytes range and very
different beasts.
E> Furthermore, how can I load that large db into cache in advance and
E> later use another perl to access/search that db?
I don't understand exactly what you want; there are 3 possibilities I
can think of and I'd rather not guess. Can you please explain what you
want to accomplish in more detail?
Ted
|
|
Posted by Ela on April 11, 2008, 11:03 am
Please log in for more thread options > I hope by "large" you mean at most a hundred gigabytes of data (which is
> considered a mid-size database by most people). Large databases are
> usually in the hundreds of gigabytes - terabytes range and very
> different beasts.
It's only 3GB... but I find when I run the current java program with the
mysql, it uses up all the 2GB memory...
>
> E> Furthermore, how can I load that large db into cache in advance and
> E> later use another perl to access/search that db?
>
> I don't understand exactly what you want; there are 3 possibilities I
> can think of and I'd rather not guess. Can you please explain what you
> want to accomplish in more detail?
>
> Ted
In MySQL, the db is already structured and therefore runtime prog can access
the structued DB. But in perl, I don't know whether I can put the DB flat
file into some data structures in advance, so users can query the DB
whenever they want instead of dynamically create the DB every time.
|
|
Posted by RedGrittyBrick on April 11, 2008, 11:40 am
Please log in for more thread options Ela wrote:
>
> It's only 3GB... but I find when I run the current java program with the
> mysql, it uses up all the 2GB memory...
3 is usually greater than 2. Unless the DBMS is very space-inefficient
at storing data and the Java program is very space-efficient at storing
data.
>> E> Furthermore, how can I load that large db into cache in advance
cache usually means memory doesn't it?
There must be something I haven't understood that explains how 3 GB data
can be expected to fit into less than 2GB of memory.
--
RGB
|
|
Posted by Ted Zlatanov on April 11, 2008, 2:12 pm
Please log in for more thread options
>> I hope by "large" you mean at most a hundred gigabytes of data (which is
>> considered a mid-size database by most people). Large databases are
>> usually in the hundreds of gigabytes - terabytes range and very
>> different beasts.
E> It's only 3GB... but I find when I run the current java program with the
E> mysql, it uses up all the 2GB memory...
You are loading all your data into memory. That's almost certainly the
wrong approach. Do you need all the data? What does the data look
like? What operations are you doing across the data? SQL has almost
every operation covered, and it's rare indeed that you will find it
inadequate for managing structured data.
E> Furthermore, how can I load that large db into cache in advance and
E> later use another perl to access/search that db?
>>
>> I don't understand exactly what you want; there are 3 possibilities I
>> can think of and I'd rather not guess. Can you please explain what you
>> want to accomplish in more detail?
E> In MySQL, the db is already structured and therefore runtime prog can access
E> the structued DB. But in perl, I don't know whether I can put the DB flat
E> file into some data structures in advance, so users can query the DB
E> whenever they want instead of dynamically create the DB every time.
I still don't understand why Perl can't query the database directly, but
you have to answer the questions above before I or anyone else can
suggest the next step without guessing.
Ted
|
| Similar Threads | Posted | | Handling Large files (a few Gb) in Perl | July 16, 2007, 7:40 am |
| Line count, the best strategy? | January 26, 2006, 12:34 am |
| Strategy about browser cache and generated pages | January 15, 2006, 9:09 pm |
| [OT] ISO book on CGI design | January 11, 2005, 11:32 pm |
| Perl web design | February 27, 2005, 5:25 am |
| CGI scripts and modular design? | September 28, 2004, 9:41 am |
| Opinions on the Design of Perl | December 13, 2004, 1:19 pm |
| When does a bug turn into a design feature? Or, what is a "bug"? | January 21, 2006, 6:50 pm |
| looking for partners and work in web design using CGI | February 3, 2006, 6:40 am |
| better design of spreadsheet generation | March 20, 2008, 7:58 am |
|
XML Sitemap