I/O with database blob

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View


My problem involves a blob which when I read it from the mysql
database, I manually read the specific data off of it so I eventually
end up with a bunch of variables with different types of data. Some of
my data is of type float.

My problem starts when I try to save my blob back into the database.

Let's say my blob contains:
$id (long)
$name (string)
$description (string)
$value (float)

When I try to save the $value element to the database, I need to put
all the pieces together to make the blob. That's where I get a

I should probably mention that I'm currently developing a php web-app
to complement an already existing c++ desktop app. That's why the
structure is as it is.

$value is equal to -680564693277057719623408366969033850880.000000
when it hasn't already been defined. So it's sort of the null value.

When I try to save -680564693277057719623408366969033850880.000000
back into the database, the database saves it as 0. Then when I get
back the value, it's actually 0 when I'd prefer having
-680564693277057719623408366969033850880.000000 .

I would like to know how to deal with this. I realize that this might
not be a very helpful explanation but that's the best I can do right

So if anyone understands where I'm coming from, let me know.



Re: I/O with database blob

dphizler wrote:
Quoted text here. Click to load it
Is the format within the blob defined by the C++? or is it that these
numbers need to be stored somehow?

I have found storing mixed data in structures almost impossible in
loosely typed languages.

Or even strongly typed ones that don't have C's flexibility in casting
or unioning.

what is the EXACT data structure (C style will do) in the blob, and is
it cast in stone?

I have to say this is where a quick php library written in C might help :-)


char *make_blob(long id, char *string, char * description, float value)


as a callable PHP function is what you need.
Assuming floats and longs are stored the same way in PHP as in C...

Re: I/O with database blob

Quoted text here. Click to load it


Thanks for the quick reply.

The structure is defined by C++ with a component called TSTREAM which
really makes things a lot easier. It seems to be cast in stone.

I actually have a make_blob (written in php) function. And I created a
class for the whole thing. So far I am able to read the data fine and
display it as well. Now I'm trying to save the data if the user
modifies it.

Unfortunately I don't feel comfortable sharing the exact structure of
the blob. But as you assumed, there are a bunch of floats, ints,
longs, objects, arrays and unions within. I'm not completely clear how
the whole thing is done but the blob seems to be saved in hexadecimal
form. That is probably contributing to my problem. The way that the C+
+ deals with the hexadecimal values and how my php code deals with it
as well.

The key I think would be to ajust the saving mechanism to deal with
floats in a more accurate way.

The C library seems like a pretty good idea, I might explore that
idea. Might be a bit complex dealing with the objects within my main

Sorry I'm not willing to share more information. I might find a
solution, it's just gonna take a bit longer.

Thanks again.


Re: I/O with database blob

dphizler wrote:
Quoted text here. Click to load it
No worries. I appreciate the confidentiality situation.

I suppose I am saying that a load of type casting between C type
structures and php is best done in C.

Re: I/O with database blob

dphizler wrote:
Quoted text here. Click to load it

You haven't given us much to go on.  No code, no information on your
database, your PHP version and OS, etc.

However, to start, your database design is terrible (violates 1NF).
Personally, I think you'll be much better off fixing the design and
correcting your C++ program than trying to fight this - you will have
continuing problems with such a design.

But to work with this problem, do you have PHP error reporting enabled
and messages displayed?  On your development system your php.ini file
should have:

error_reporting=E_ALL // or E_ALL | E_STRICT

Do you get any messages with the above?

What happens when you try to print this value?  How do you "put the
pieces back together"?  What do you get if you try to print the value
after you've put the pieces back together?

Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.

Re: I/O with database blob

Quoted text here. Click to load it


To ease your worry. The database designed is optimized and works fine.
The ID is saved outside of the blob as well. Maybe it shouldn't be in
the blob too but either way, it's there.

I will probably try checking out the errors or notices that arise from
my code. Thanks.

As for not including any code. Well it doesn't seem to work well yet
so, I think I just want to get a few general ideas to continue my
debugging because I've been unsuccessful so far. But I will try to be
more elaborate once I have things more handled.

Quoted text here. Click to load it
When I print value after being read from the database I get
-680564693277057719623408366969033850880.000000 when it's empty but in
the C++ application it shows as -NAN (which means not a number)

Quoted text here. Click to load it
I use a bunch of functions that encode the data back into hexadecimal
format. These functions work because someone else managed to save data
in the same way as I am doing within this database. I'm just working
other another type of object. (long story)

Quoted text here. Click to load it
It's lots of hexadecimal data which is unreadable. But when I try to
decode it again, I get $value = 0 which isn't quite what I had
initially. But the whole process works fine for data that is actually
a number.

Anyway, you have given me a few pointers in your post and thank you
for that.


Re: I/O with database blob

Quoted text here. Click to load it
Quoted text here. Click to load it

How do you figure that, Jerry? I think you're assuming facts not in evidence.

Re: I/O with database blob

On Thu, 08 Jul 2010 01:06:20 GMT, in comp.lang.php spambait@milmac.com
(Doug Miller)

Quoted text here. Click to load it

If the OP is using the correct terminology then I read that they are
concatenation a series of data items and storing it into a blob data
type field. In this case - yes it is poor db design.

Re: I/O with database blob

Quoted text here. Click to load it
It really depends on what you want to do with the database. You can't
assume that from the get-go. I said it and will say it again, the
database is well designed for the needs we have for it.

Re: I/O with database blob

dphizler wrote:
Quoted text here. Click to load it

Not at all.  It works in C++.  But you're having problems when you
switch to another language.

Such things occur when you have a poor database design.

Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.

Re: I/O with database blob

On Wed, 7 Jul 2010 19:55:28 -0700 (PDT), in comp.lang.php dphizler

Quoted text here. Click to load it

Whatever <sigh>

Has the thought crossed your mind that maybe the language that you are
using can't cope with such a large float value? Have you investigated

Re: I/O with database blob

On 08/07/10 03:55, dphizler wrote:
Quoted text here. Click to load it

Clearly not, as you're having a problem accessing the data in it.

I know this isn't comp.lang.c++, but if Jerry will just bear with me ....

One day, you might get a new compiler, maybe because you're working on a
new platform with say 128 bits instead of 64 or 32 bits, and your new
compiler will probably store your struct differently. When that happens,
your current database files will be hosed, because the stored structs
will no longer correspond to the structs that the compiler creates in
the new code.

A struct is basically a record containing fields. So is a database row.
If you want to store the data from your structs in a database, it would
make a lot more sense to store the individual fields, rather than the
binary representation of the struct.

a. It's more future proof.
b. It's easier to interface to other applications sharing the database.


Denis McMahon

Re: I/O with database blob

.oO(Denis McMahon)

Quoted text here. Click to load it

Such a structure should be well-defined in a way that every compiler
generates the same results, regardless of the hardware. Many binary file
formats (e.g. images) contain float values, and obviously they still
work on every platform available.

And the same problem may arise with integers, but it doesn't - a 32bit
integer is a 32bit integer, even on a 64bit platform.

Quoted text here. Click to load it

Of course the required size of the fields should be defined.

Quoted text here. Click to load it

Would you also store JPEG header fields in single DB fields, separated
from the image data? Obviously not.

Quoted text here. Click to load it

We don't know what the format is all about, where it's coming from and
what applications are supposed to work with it. So it's quite possible
that it makes sense for them to store it as a whole, like an image or
any other well-defined binary data structure. No one would split such
data when it's stored in a DB.


Re: I/O with database blob

On Fri, 09 Jul 2010 00:25:13 +0200, in comp.lang.php Michael Fesser

Quoted text here. Click to load it

How do you define an integer in C++?
How do you define an integer in PHP?
Quoted text here. Click to load it

How do you type cast variables in a non type casted language?
Quoted text here. Click to load it

Except when the data structure works with a strongly type cast
language and then the platform changes to a non type casted language.

Re: I/O with database blob

On 08/07/10 23:25, Michael Fesser wrote:

Quoted text here. Click to load it

Yes, but the storage of each element of data within the struct may be on
processor word boundaries (this is a compiler matter).

Defining the field sizes doesn't define the padding between fields. When
you write the struct out as a binary object, you get the padding as
well, and that can vary from compiler to compiler.

Now, this doesn't matter when you're just handling structs in memory
within the program that has defined them, but when you start passing
them around between different applications, whether by way of files or
by way of blobs in a database, it can become an issue.

In fact, it's been documented a few times that different compilers can
not simply exchange binary representations of structs by way of eg files
for this very reason!

If you write a struct out to some external (to the program that creates
it) storage (eg a file or a database as a blob) as a binary chunk of
data, and later try and read it with the same program compiled for a
different architecture, eg 128 bit instead of 64 bit, you might find
that what is read doesn't match up with what was written, because each
element in your old struct was actually padded to multiples of 64 bits,
but your new code expects each element of the struct to be padded to 128

so eg if I have a struct of 4 x 32 bit integers and 4 x unsigned char on
a 32 bit platform, they might each take up 4 bytes, with the characters
having 3 bytes of padding to align them on a 32 bit word boundary.

The struct size is 32 bytes.

If I now compile the code on a 128 bit processor, each element might now
be 16 bytes. The ints might have 12 bytes of padding each, and the chars
might have 15 bytes of padding each, so that each element of the struct
is aligned on a memory word boundary.

The struct size is now 128 bytes. The struct is still 4 x 32 bit ints
and 4 x unsigned char, but the way it's stored is different.

If I try and save the binary representation of the 32 bit architecture
compiled struct in a database, and then read it with the same code
compiled for 128 bit architecture, I may get problems because although
the data representations are the same, the binary storage of the struct
is different.

See also this faq question amongst others:



Denis McMahon

Re: I/O with database blob

Denis McMahon wrote:
Quoted text here. Click to load it

Which contains the one statement that makes nonsense of all the above

"If the layout of the structure cannot be changed (e.g., when it must
match some external specification, like a block of data returned by a
system call), you can use the __attribute__((packed)) extension of GCC
(see GNU C/C++ Manual.) to prevent GCC from padding the structure
members; this will make accesses to some of the members significantly
slower. "


Quoted text here. Click to load it

Re: I/O with database blob

On 09/07/10 16:38, The Natural Philosopher wrote:

Quoted text here. Click to load it

That, of course, is assuming that

(a) either gcc with the packed attribute or it's equivalent (if one was
available) was used for the original code; and

(b) that whatever compilers are used to recompile the code for any
future systems have a similar mechanism and it works in the same way.

So no, it doesn't make nonsense of all the above, because it's a
specific mechanism in one compiler.

It might not even work if the compiler that created the data created a
padded struct, because if you try and use a packed struct, it will be a
different size.

The advantage of having packed structs is that you can use single
characters of padding to try and map the meaningful data when you do


struct {
 int a, b;
 char c, d;

padded on a 32 bit system this might be stored like:


where p is padding.

saved as a blob this becomes:


To read this into a 128 bit system, you could use the packed struct:

struct {
 int a, b;
 char z, y, x, c, w, v, u, d;

However you'd still need to re-write the underlying code to account for
any padding in the original structs.

Your "magic solution" only works if the current data is packed, if it's
not, then at some point a conversion is still going to be needed.

Storing binary representations of structs is causing problems for the OP
now, cf his problems getting php to read and write them, it would make
more sense, if their code wants to store those structs in a database, to
store the elements of the structs in a table that had as it's fields the
elements of the struct.


Denis McMahon

Re: I/O with database blob

Denis McMahon wrote:
Quoted text here. Click to load it

Nope. Its simply one of the things you would check porting to a new
compiler or machine word length.

Quoted text here. Click to load it
If not,. throw and error: Its them up to your successor to figure out
how to get his shiny new compiler to work.

If it were that simple you wouldn't get paid to be a software engineer ;-)

Quoted text here. Click to load it

Other mechanisms exist to do similar.

You talk as if this was a Big Issue. We have had to cope with it for
YEARS. ever since Motorola and Intel went different ways as to whether
MSB was high byte or low byte.

At the end of the day you could make a pack function or macro.

Quoted text here. Click to load it

It might even not work if you stood on your head and pissed in the disk


Im bored with this: code portability is something you learn how to do IF
you have to cross compile to different machines. Either you have done
it, and its no big deal, just tedious and boring, or you haven't, and
want to get all worked up about it.

Re: I/O with database blob

Denis McMahon wrote:
Quoted text here. Click to load it

That's when you redefine your struct{} statements more closely.

Re: I/O with database blob

On 09/07/10 00:10, The Natural Philosopher wrote:
Quoted text here. Click to load it

You're still hosed, because your new machine is looking for 128 bit
alignment, so what you have to do is write a conversion app to change
all the 64 or 32 bit aligned struct data in the old binary objects in
the database to new style 128 bit aligned data.

You see, the new compiler doesn't care what you define in your struct,
it aligns every struct member of a 128 bit word boundary.

But your old struct was based on a 64 bit word boundary, or even a 32
bit word boundary.

So your old struct had 4 ints a through d, and these were stored in 128
bits (16 bytes) as

aaaa - 32 bit word 0
bbbb - 32 bit word 1
cccc - 32 bit word 2
dddd - 32 bit word 3

when you write this to a file or database as a binary object you get 16
bytes: aaaabbbbccccdddd

But your new 128 bit memory model stores the same 4 int struct as:

------------aaaa - 128 bit word 0
------------bbbb - 128 bit word 1
------------cccc - 128 bit word 2
------------dddd - 128 bit word 3

(where ------------ = 12 bytes of padding)

and when it reads in a binary struct written by your old 32 bit code, it

aaaabbbbccccdddd - 128 bit word 0
xxxxxxxxxxxxxxxx - 128 bit word 1
xxxxxxxxxxxxxxxx - 128 bit word 2
xxxxxxxxxxxxxxxx - 128 bit word 3

where x is an undefined value

Now, it might, knowing that it's looking for a 32 bit int, be smart
enough to give you dddd when you ask for the first int value in the
struct that was read, but as it looks for 128 bit aligned values, it
will never read the 32 bit aligned ones correctly.

Fortunately in this case you can write a conversion routine that reads a
single 128 bit int and converts it to 4 32 bit ints by bit shifting and

However, supposing your struct was 7 x 32 bit ints. Now your old struct
size doesn't even map to a valid struct size in your new architecture.

Fortunately you can probably read the binary data as an arbitrary length
array of unsigned char, and then walk along the resulting 'string' to
extract individual characters into their original struct members to
recreate the values of those members.

However, it's bloody tedious doing so, which is why storing binary
representations of internal storage structures is "a very very bad idea".

As is being shown by the difficulty the op is having in using those
binary representations of c storage structures that have been saved in a
database which he now wishes to access with php.

If you're going to put a c struct in a database, create a table with
columns for each struct member and store each struct as a database record.


Denis McMahon

Site Timeline