BerkeleyDB Queue Database Array Size

Home | NewsGroups | Search | About

comp.lang.perl.modules

get this group's latest topics as an RSS feed

add this group's latest topics to your My MSN content

add this group's latest topics to your My Yahoo content

Subject

Author

Date

BerkeleyDB Queue Database Array Size

kallestad

08-14-2006

Re: BerkeleyDB Queue Database Array Size

Paul Marquess

08-14-2006

Re: BerkeleyDB Queue Database Array Size

Paul Marquess

08-14-2006

Re: BerkeleyDB Queue Database Array Size

Paul Marquess

08-14-2006

Re: BerkeleyDB Queue Database Array Size

Paul Marquess

08-14-2006

Re: BerkeleyDB Queue Database Array Size

kallestad

08-14-2006

Posted by kallestad on August 14, 2006, 2:50 am

Please log in for more thread options

I've been playing around with BerkeleyDB Queue database type, and I've
found something curious.

I'm tying an array variable to the queue database:
$db = tie @samplearray, 'BerkeleyDB::Queue',
-Flags => DB_CREATE,
-Filename => $config->get('db_queue_filename'),
-Env => $env,
-Len => 300
or print "$BerkeleyDB::Error\n" ;

I'm adding records to the database by pushing values into the array:
push @samplearray, $u or die "can't push $u $!\n";

And I'm pulling records from the beginning of the database using shift:
shift @samplearray;

Everything is working brilliantly - although I haven't tried
multiprocess access to the database which is a desired future step.
The curiosity that I have found is this:

$k = $#samplearray;
print "$k \n";

When the array is not "tied" to the BerkeleyDB, this number will shrink
every time I shift a record out of the array - If I pull out 5 of 5
items, it would report 3,2,1,0,-1. However, when it is tied to the
database, it will always report the maximum queue size until there is
nothing left in the queue, at which point it will return -1 like it
normally would if it was just an array. In the above example - it's
4,4,4,4,-1

I tried calling $db->sync after pulling items out, but the issue still
exists. Being a bit of a newbie, I don't know if this is a problem
with the way I am implementing things, my technical understanding, or
if this is an actual bug. Any guidance is appreciated.

Thanks,
Steve

Posted by Paul Marquess on August 14, 2006, 4:38 am

Please log in for more thread options

: 7bit

> -----Original Message-----

> From: kallestad@gmail.com [mailto:kallestad@gmail.com]

> Posted At: 14 August 2006 07:50

> Posted To: comp.lang.perl.modules

> Conversation: BerkeleyDB Queue Database Array Size

> Subject: BerkeleyDB Queue Database Array Size

> I've been playing around with BerkeleyDB Queue database type, and I've

> found something curious.

> I'm tying an array variable to the queue database:

> $db = tie @samplearray, 'BerkeleyDB::Queue',

> -Flags => DB_CREATE,

> -Filename => $config->get('db_queue_filename'),

> -Env => $env,

> -Len => 300

> or print "$BerkeleyDB::Error\n" ;

> I'm adding records to the database by pushing values into the array:

> push @samplearray, $u or die "can't push $u $!\n";

> And I'm pulling records from the beginning of the database using

shift:

> shift @samplearray;

> Everything is working brilliantly - although I haven't tried

> multiprocess access to the database which is a desired future step.

> The curiosity that I have found is this:

> $k = $#samplearray;

> print "$k \n";

> When the array is not "tied" to the BerkeleyDB, this number will

shrink

> every time I shift a record out of the array - If I pull out 5 of 5

> items, it would report 3,2,1,0,-1. However, when it is tied to the

> database, it will always report the maximum queue size until there is

> nothing left in the queue, at which point it will return -1 like it

> normally would if it was just an array. In the above example - it's

> 4,4,4,4,-1

> I tried calling $db->sync after pulling items out, but the issue still

> exists. Being a bit of a newbie, I don't know if this is a problem

> with the way I am implementing things, my technical understanding, or

> if this is an actual bug. Any guidance is appreciated.

> Thanks,

> Steve

------=_NextPart_000_0068_01C6BF85.6FA02680
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7036.0">
<TITLE>Re: BerkeleyDB Queue Database Array Size</TITLE>
</HEAD>
<BODY>

 
 

> -----Original Message-----

 > From: kallestad@gmail.com [<A =

 > Posted At: 14 August 2006 07:50

 > Posted To: comp.lang.perl.modules

 > Conversation: BerkeleyDB Queue Database Array =
Size

 > Subject: BerkeleyDB Queue Database Array =
Size

 > 

 > I've been playing around with BerkeleyDB Queue =
database type, and I've

 > found something curious.

 > 

 > I'm tying an array variable to the queue =
database:

 > $db =3D tie @samplearray, =
'BerkeleyDB::Queue',

 >        =
-Flags    =3D> DB_CREATE,

 >        =
-Filename =3D> $config->get('db_queue_filename'),

 >       =
     -Env      =3D> =
$env,

 >       =
     -Len      =3D> =
300

 >          =
  or print "$BerkeleyDB::Error\n" ;

 > 

 > I'm adding records to the database by pushing =
values into the array:

 > push @samplearray, $u or die "can't push $u =
$!\n";

 > 

 > And I'm pulling records from the beginning of =
the database using shift:

 > 

 > Everything is working brilliantly - although I =
haven't tried

 > multiprocess access to the database which is a =
desired future step.

 > The curiosity that I have found is this:

 > 

 >  $k =3D $#samplearray;

 >  print "$k \n";

 > 

 > When the array is not "tied" to the =
BerkeleyDB, this number will shrink

 > every time I shift a record out of the array - =
If I pull out 5 of 5

 > items, it would report 3,2,1,0,-1.  =
However, when it is tied to the

 > database, it will always report the maximum =
queue size until there is

 > nothing left in the queue, at which point it =
will return -1 like it

 > normally would if it was just an array.  In =
the above example - it's

 > 4,4,4,4,-1

 > 

 > I tried calling $db->sync after pulling items =
out, but the issue still

 > exists.  Being a bit of a newbie, I don't =
know if this is a problem

 > with the way I am implementing things, my =
technical understanding, or

 > if this is an actual bug.  Any guidance is =
appreciated.

 > 

 > Thanks,

 > Steve


</BODY>
</HTML>
------=

Posted by Paul Marquess on August 14, 2006, 5:00 am

Please log in for more thread options

: 7bit

From: kallestad@gmail.com [mailto:kallestad@gmail.com]

> I've been playing around with BerkeleyDB Queue database type, and I've

> found something curious.

> I'm tying an array variable to the queue database:

> $db = tie @samplearray, 'BerkeleyDB::Queue',

> -Flags => DB_CREATE,

> -Filename => $config->get('db_queue_filename'),

> -Env => $env,

> -Len => 300

> or print "$BerkeleyDB::Error\n" ;

> I'm adding records to the database by pushing values into the array:

> push @samplearray, $u or die "can't push $u $!\n";

> And I'm pulling records from the beginning of the database using

shift:

> shift @samplearray;

> Everything is working brilliantly - although I haven't tried

> multiprocess access to the database which is a desired future step.

> The curiosity that I have found is this:

> $k = $#samplearray;

> print "$k \n";

> When the array is not "tied" to the BerkeleyDB, this number will

shrink

> every time I shift a record out of the array - If I pull out 5 of 5

> items, it would report 3,2,1,0,-1. However, when it is tied to the

> database, it will always report the maximum queue size until there is

> nothing left in the queue, at which point it will return -1 like it

> normally would if it was just an array. In the above example - it's

> 4,4,4,4,-1

> I tried calling $db->sync after pulling items out, but the issue still

> exists. Being a bit of a newbie, I don't know if this is a problem

> with the way I am implementing things, my technical understanding, or

> if this is an actual bug. Any guidance is appreciated.

This is a limitation in the current perl interface to Berkeley DB.

The problem with providing an interface to return the number of records
in the queue (or any of the other database types that Berkeley DB
provides) is that the underlying Berkeley DB library has to traverse the
complete database to get an accurate count. If you have a large queue,
this is not desirable.

Have a look at this page. In particular, look at the DB_FAST_STAT entry.

http://www.sleepycat.com/docs/api_c/db_stat.html

If the limitations of DB_FAST_STAT are good enough for your purposes,
you can use it like this

$count = $db->db_stat("qs_nrecs", DB_FAST_STAT);

If you want to take the hit of traversing the database to count the
number of records, this should do the trick

$count = $db->db_stat("qs_nrecs");

Paul

------=_NextPart_000_0071_01C6BF88.6AFC60A0
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7036.0">
<TITLE>Re: BerkeleyDB Queue Database Array Size</TITLE>
</HEAD>
<BODY>


From: kallestad@gmail.com [<A =

  

 > I've been playing around with BerkeleyDB Queue =
database type, and I've

 > found something curious.

 > 

 > I'm tying an array variable to the queue =
database:

 > $db =3D tie @samplearray, =
'BerkeleyDB::Queue',

 >        =
-Flags    =3D> DB_CREATE,

 >        =
-Filename =3D> $config->get('db_queue_filename'),

 >       =
     -Env      =3D> =
$env,

 >       =
     -Len      =3D> =
300

 >          =
  or print "$BerkeleyDB::Error\n" ;

 > 

 > I'm adding records to the database by pushing =
values into the array:

 > push @samplearray, $u or die "can't push $u =
$!\n";

 > 

 > And I'm pulling records from the beginning of =
the database using shift:

 > 

 > Everything is working brilliantly - although I =
haven't tried

 > multiprocess access to the database which is a =
desired future step.

 > The curiosity that I have found is this:

 > 

 >  $k =3D $#samplearray;

 >  print "$k \n";

 > 

 > When the array is not "tied" to the =
BerkeleyDB, this number will shrink

 > every time I shift a record out of the array - =
If I pull out 5 of 5

 > items, it would report 3,2,1,0,-1.  =
However, when it is tied to the

 > database, it will always report the maximum =
queue size until there is

 > nothing left in the queue, at which point it =
will return -1 like it

 > normally would if it was just an array.  In =
the above example - it's

 > 4,4,4,4,-1

 > 

 > I tried calling $db->sync after pulling items =
out, but the issue still

 > exists.  Being a bit of a newbie, I don't =
know if this is a problem

 > with the way I am implementing things, my =
technical understanding, or

 > if this is an actual bug.  Any guidance is =
appreciated.


This is a limitation in the current perl interface to =
Berkeley DB. 


The problem with providing an interface to return the =
number of records in the queue (or any of the other database types that =
Berkeley DB provides) is that the underlying Berkeley DB library has to =
traverse the complete database to get an accurate count. If you have a =
large queue, this is not desirable.

Have a look at this page. In particular, look at the =
DB_FAST_STAT entry.


  <A =
HREF=3D"http://www.sleepycat.com/docs/api_c/db_stat.html">http://www.slee=
pycat.com/docs/api_c/db_stat.html</A> 


If the limitations of DB_FAST_STAT are good enough for =
your purposes, you can use it like this

 

  $count =3D =
$db->db_stat("qs_nrecs", DB_FAST_STAT);


If you want to take the hit of traversing the database =
to count the number of records, this should do the trick


  $count =3D =
$db->db_stat("qs_nrecs");


Paul


</BODY>
</HTML>
------=

Posted by Paul Marquess on August 14, 2006, 10:38 am

Please log in for more thread options

: 7bit

From: kallestad@gmail.com [mailto:kallestad@gmail.com]

> I've been playing around with BerkeleyDB Queue database type, and I've

> found something curious.

> I'm tying an array variable to the queue database:

> $db = tie @samplearray, 'BerkeleyDB::Queue',

> -Flags => DB_CREATE,

> -Filename => $config->get('db_queue_filename'),

> -Env => $env,

> -Len => 300

> or print "$BerkeleyDB::Error\n" ;

> I'm adding records to the database by pushing values into the array:

> push @samplearray, $u or die "can't push $u $!\n";

> And I'm pulling records from the beginning of the database using

shift:

> shift @samplearray;

> Everything is working brilliantly - although I haven't tried

> multiprocess access to the database which is a desired future step.

> The curiosity that I have found is this:

> $k = $#samplearray;

> print "$k \n";

> When the array is not "tied" to the BerkeleyDB, this number will

shrink

> every time I shift a record out of the array - If I pull out 5 of 5

> items, it would report 3,2,1,0,-1. However, when it is tied to the

> database, it will always report the maximum queue size until there is

> nothing left in the queue, at which point it will return -1 like it

> normally would if it was just an array. In the above example - it's

> 4,4,4,4,-1

> I tried calling $db->sync after pulling items out, but the issue still

> exists. Being a bit of a newbie, I don't know if this is a problem

> with the way I am implementing things, my technical understanding, or

> if this is an actual bug. Any guidance is appreciated.

This is a limitation in the current perl interface to Berkeley DB.

The problem with providing an interface to return the number of records
in the queue (or any of the other database types that Berkeley DB
provides) is that the underlying Berkeley DB library has to traverse the
complete database to get an accurate count. If you have a large queue,
this is not desirable.

Have a look at this page. In particular, look at the DB_FAST_STAT entry.

http://www.sleepycat.com/docs/api_c/db_stat.html

If the limitations of DB_FAST_STAT are good enough for your purposes,
you can use it like this

$count = $db->db_stat("qs_nrecs", DB_FAST_STAT);

If you want to take the hit of traversing the database to count the
number of records, this should do the trick

$count = $db->db_stat("qs_nrecs");

Paul

------=_NextPart_000_00D6_01C6BFB7.AA2241D0
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7036.0">
<TITLE>Re: BerkeleyDB Queue Database Array Size</TITLE>
</HEAD>
<BODY>


From: kallestad@gmail.com [<A =

> I've been playing around with BerkeleyDB Queue =
database type, and I've

 > found something curious.

 > 

 > I'm tying an array variable to the queue =
database:

 > $db =3D tie @samplearray, =
'BerkeleyDB::Queue',

 >        =
-Flags    =3D> DB_CREATE,

 >        =
-Filename =3D> $config->get('db_queue_filename'),

 >       =
     -Env      =3D> =
$env,

 >       =
     -Len      =3D> =
300

 >          =
  or print "$BerkeleyDB::Error\n" ;

 > 

 > I'm adding records to the database by pushing =
values into the array:

 > push @samplearray, $u or die "can't push $u =
$!\n";

 > 

 > And I'm pulling records from the beginning of =
the database using shift:

 > 

 > Everything is working brilliantly - although I =
haven't tried

 > multiprocess access to the database which is a =
desired future step.

 > The curiosity that I have found is this:

 > 

 >  $k =3D $#samplearray;

 >  print "$k \n";

 > 

 > When the array is not "tied" to the =
BerkeleyDB, this number will shrink

 > every time I shift a record out of the array - =
If I pull out 5 of 5

 > items, it would report 3,2,1,0,-1.  =
However, when it is tied to the

 > database, it will always report the maximum =
queue size until there is

 > nothing left in the queue, at which point it =
will return -1 like it

 > normally would if it was just an array.  In =
the above example - it's

 > 4,4,4,4,-1

 > 

 > I tried calling $db->sync after pulling items =
out, but the issue still

 > exists.  Being a bit of a newbie, I don't =
know if this is a problem

 > with the way I am implementing things, my =
technical understanding, or

 > if this is an actual bug.  Any guidance is =
appreciated.


This is a limitation in the current perl interface to =
Berkeley DB. 


The problem with providing an interface to return the =
number of records in the queue (or any of the other database types that =
Berkeley DB provides) is that the underlying Berkeley DB library has to =
traverse the complete database to get an accurate count. If you have a =
large queue, this is not desirable.

Have a look at this page. In particular, look at the =
DB_FAST_STAT entry.


  <A =
HREF=3D"http://www.sleepycat.com/docs/api_c/db_stat.html">http://www.slee=
pycat.com/docs/api_c/db_stat.html</A> 


If the limitations of DB_FAST_STAT are good enough for =
your purposes, you can use it like this

 

  $count =3D =
$db->db_stat("qs_nrecs", DB_FAST_STAT);


If you want to take the hit of traversing the database =
to count the number of records, this should do the trick


  $count =3D =
$db->db_stat("qs_nrecs");


Paul


</BODY>
</HTML>
------=

Posted by Paul Marquess on August 14, 2006, 12:06 pm

Please log in for more thread options

kallestad@gmail.com wrote:

> I've been playing around with BerkeleyDB Queue database type, and I've

> found something curious.

> I'm tying an array variable to the queue database:

> $db = tie @samplearray, 'BerkeleyDB::Queue',

> -Flags => DB_CREATE,

> -Filename => $config->get('db_queue_filename'),

> -Env => $env,

> -Len => 300

> or print "$BerkeleyDB::Error\n" ;

> I'm adding records to the database by pushing values into the array:

> push @samplearray, $u or die "can't push $u $!\n";

> And I'm pulling records from the beginning of the database using shift:

> shift @samplearray;

> Everything is working brilliantly - although I haven't tried

> multiprocess access to the database which is a desired future step.

> The curiosity that I have found is this:

> $k = $#samplearray;

> print "$k \n";

> When the array is not "tied" to the BerkeleyDB, this number will shrink

> every time I shift a record out of the array - If I pull out 5 of 5

> items, it would report 3,2,1,0,-1. However, when it is tied to the

> database, it will always report the maximum queue size until there is

> nothing left in the queue, at which point it will return -1 like it

> normally would if it was just an array. In the above example - it's

> 4,4,4,4,-1

> I tried calling $db->sync after pulling items out, but the issue still

> exists. Being a bit of a newbie, I don't know if this is a problem

> with the way I am implementing things, my technical understanding, or

> if this is an actual bug. Any guidance is appreciated

Similar Threads	Posted
Cant load ...BerkeleyDB.so for module BerkeleyDB: libdb-4.4.so	January 19, 2006, 1:53 pm
MIME::Lite need sendmail Queue ID	November 16, 2006, 9:34 am
No queue support with the Sleepycat::Db Perl interface?	March 28, 2006, 7:03 pm
[RFC] Thread::MappedQueue - threadsafe queue with mapped responses	July 8, 2005, 11:47 pm
Proc::ProcessTable size differences	November 7, 2004, 12:17 am
Term-Size make fails on debian	September 6, 2005, 4:28 pm
help with module naming: filehandle-tie for size-limited/rotating logfile	March 18, 2007, 5:44 am
Object oriented database	February 8, 2008, 10:37 am
ANNOUNCE: Set::Array V 0.13	April 9, 2006, 12:25 am
ANNOUNCE: Set::Array V 0.14	April 10, 2006, 6:11 am

Our other projects:

Art Dolls, Fairies and Mermaids - Sunnyfaces.net

Roy's Linux, Programming and Search Engines messages

1-Script XML Sitemap

XML Sitemap