Restore v1.0 XML backup in to Gallery 2.3 which expects v1.1 ?

libove

Joined: 2007-08-30
Posts: 69
Posted: Fri, 2010-01-29 08:54

I am trying to restore an old, XML style backup of my database. I have the full g2data directory, but the database became corrupt and is much older than the g2data directory, and is in XML format, v1.0. Here is the core module version information from the very top of the XML backup file:

<?xml version="1.0"?>
<export schema-version="1.0" tables="45" core-version="1.2.21" gallery-version="2.3-svn">

I have a brand new Gallery2 v2.3.1 (core 1.3.0.1) installation in to which I want to restore this XML backup.

Executing the gallery2/lib/support/index.php?import function, selecting the .xml backup file, I get these errors:

Errors Occurred
The exported file is not the correct version. Expecting 1.1 received 1.0.
The available version of plugin "albumselect" may not be compatible with the imported version.
The available version of plugin "archiveupload" may not be compatible with the imported version.
The available version of plugin "comment" may not be compatible with the imported version.
The available version of plugin "publishxp" may not be compatible with the imported version.
The available version of plugin "webdav" may not be compatible with the imported version.

Is it possible to import this XML file in to this Gallery2 version? .. or must I install an appropriately old Gallery2 version, import, then upgrade Gallery2?
Which version of Gallery2 will best support this XML file, based on that odd "2.3-svn" version string in the XML backup?

Thanks,
Jay

Here's the rest of the system information of the new 2.3.1 installation:

Gallery URL = http://gator1173.hostgator.com/~libove/gallery2/main.php
Gallery version = 2.3.1 core 1.3.0.1
API = Core 7.54, Module 3.9, Theme 2.6, Embed 1.5
PHP version = 5.2.12 cgi-fcgi
Webserver = Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
Database = mysqli 5.1.30, lock.system=flock
Toolkits = ArchiveUpload, Exif, Getid3, ImageMagick, LinkItemToolkit, SquareThumb, Thumbnail, Gd
Acceleration = none, none
Operating system = Linux gator1173.hostgator.com 2.6.28.9 #51 SMP Fri Jan 15 22:59:44 CST 2010 i686
Default theme = matrix
gettext = enabled
Locale = en_US
Browser = Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; FDM; MS-RTC LM 8)
Rows in GalleryAccessMap table = 32
Rows in GalleryAccessSubscriberMap table = 3
Rows in GalleryUser table = 2
Rows in GalleryItem table = 1
Rows in GalleryAlbumItem table = 1
Rows in GalleryCacheMap table = 0

 
alecmyers

Joined: 2006-08-01
Posts: 4342
Posted: Fri, 2010-01-29 11:26
Quote:
but the database ... is much older than the g2data directory

That probably means whatever you do will never achieve a satisfactory result. The two need to match exactly for a backup to be any use.

I would install from scratch and reimport your images; you will have to recreate all the rest of the data.

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Wed, 2010-09-22 18:19

I wrote a script for libove that compares filenames and size from the old xml backup to items in the new gallery.
It then copies the old title, summary, description and keywords over to the matched gallery item.
All in about ~120 lines of code. Not bad.

If anyone is interested, lemme know.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
libove

Joined: 2007-08-30
Posts: 69
Posted: Wed, 2010-09-22 18:30

The amount of time and help which suprsidr has been giving me is so far above and beyond, I must have accumulated a huge karmic debt :)
I sure hope he's having fun with this!

I hope I can give back somehow. I'm not a PHP coder (my last coding was more than a decade ago in "C", yeah, I'm an old fart). I'm well qualified in security and privacy governance, in case the project ever needs some advice on how to structure things to make it easier for a user of the gallery to comply with laws and regulations relating to infosec, privacy or data protection anywhere in the world.

-Jay

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Tue, 2010-11-09 20:33

suprsidr, I think I have the samen problem you faced. After a upgrade from G2.2.4 to G.2.3.1 I screwed up the database. I have however a xml export of the database that was made during the upgrade wizard. I now want to import / recover this database. Do you know how? I described the problem more in detail in post http://gallery.menalto.com/node/99037#comment-359402

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Tue, 2010-11-09 23:42

yeah, I never faced said problem.
But as my post above states, I wrote a script to help fix this issue - if you're interested.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Tue, 2010-11-09 23:51

I am definitly intrested! Almost 10 years of photo's are about to be lost.

So this script may help my situation? I thus have an xml export of my 2.2.4 database and the g2data folder. I also have a corrupted installed gallery 2.3.1 (because I accidently deleted the database). Ofcourse I can re-install a fresh copy of G2.3.1 or, if needed 2.2.4 (I have the original zip)

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Wed, 2010-11-10 00:50

You'll need to install G2.3, and do a server import to get all your images back into gallery.
Then follow the directions in the top of the script.

This script compares images by filename and size and copies the relevant data like title, description ... it won't copy comments and such as that's a separate module w/ its own terms.

Running on such a larger gallery may need some tweaking.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Wed, 2010-11-10 09:59

thanks. I am going to give it a try!

Currently the g2data is on my local windows server. Should I do the import on this local server only, or is it possible to install G2.3 on a remote linux hosting server, transfer the g2data and xml export there and then do an import? I prefer the latter, because my local server is going to have a though time making for thousands of pics thumbnails. I however don't know if this will give problems since your tool depends on filename and size (perhaps the size is different on linux filesystem because of minimum chunk size?).

What kind of tweaking will be needed? I believe the amount of imported pictures are somewhere between 5000 - 10.000.

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Wed, 2010-11-10 12:20

Gallery will recreate albums based on directory structure of your old albums directory

then rum my script to transfer title, description...

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Wed, 2010-11-10 23:31

hmmm....I ran the Add Item as in the URL you provided, but get the following error (everything is chmod 755).

Quote:
Adding items

Estimated time remaining: 13:50

Memory used: 23506520, total: 128M

Error (ERROR_STORAGE_FAILURE) : Empty sequence id!

* in modules/core/classes/GalleryStorage/GalleryStorageExtras.class at line 494 (GalleryCoreApi::error)
* in modules/core/classes/GalleryStorage.class at line 1872 (GalleryStorageExtras::_getUniqueIdWithConnection)
* in modules/core/classes/GalleryStorage/GalleryStorageExtras.class at line 446 (MySqlStorage::getUniqueId)
* in modules/core/classes/GalleryStorage.class at line 347 (GalleryStorageExtras::newEntity)
* in modules/core/classes/GalleryEntity.class at line 168 (GalleryStorage::newEntity)
* in modules/core/classes/GalleryChildEntity.class at line 69 (GalleryEntity::create)
* in modules/core/classes/GalleryFileSystemEntity.class at line 93 (GalleryChildEntity::create)
* in modules/core/classes/GalleryItem.class at line 192 (GalleryFileSystemEntity::create)
* in modules/core/classes/GalleryDataItem.class at line 141 (GalleryItem::create)
* in modules/core/classes/GalleryPhotoItem.class at line 104 (GalleryDataItem::create)
* in modules/core/classes/helpers/GalleryItemHelper_medium.class at line 148 (GalleryPhotoItem::create)
* in modules/core/classes/GalleryCoreApi.class at line 1668 (GalleryItemHelper_medium::addItemToAlbum)
* in modules/itemadd/ItemAddFromServer.inc at line 313 (GalleryCoreApi::addItemToAlbum)
* in modules/itemadd/ItemAddFromServer.inc at line 422 (ItemAddFromServer::addItem)
* in modules/itemadd/ItemAddFromServer.inc at line 476 (ItemAddFromServer::processDirectory)
* in modules/itemadd/ItemAddFromServer.inc at line 264 (ItemAddFromServer::processDirectory)
* in modules/core/ItemAdd.inc at line 159 (ItemAddFromServer::handleRequest)
* in ??? at line 0 (ItemAddController::handleRequestWithProgressBar)
* in modules/core/classes/GalleryTemplateAdapter.class at line 1089
* in /home/mehranl/g2data/smarty/templates_c/%%3775568552/matrix/%%3A^3A8^3A818B59%%theme.tpl.php at line 75 (GalleryTemplateAdapter::trailer)
* in lib/smarty/Smarty.class.php at line 1255
* in modules/core/classes/GallerySmarty.class at line 61 (Smarty::fetch)
* in modules/core/classes/GalleryTemplate.class at line 241 (GallerySmarty::fetch)
* in main.php at line 506 (GalleryTemplate::display)
* in main.php at line 104
* in main.php at line 88

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Wed, 2010-11-10 23:49

Add from local server requires 777 recursively on your g2data and I believe the old album directory as well.

Luckily you can delete the few that may have been created, and start again until you get it right.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Thu, 2010-11-11 00:25

the FAQ said that 755 should be sufficient, but I will do a recursive 777 just to be sure.

I have about 9000 pictures to add (17 Gb). I read in some post my host might be killing processes that take up too much time / memory and that creating thumbnails can be shutt of, and to use of symbolic links. Will your script work if I will do this?

<edit>my intension is, if I get this working, I want to upgrade to g3. So the symbolic links should mess up the possibility to later to a g2 import into g3</edit>

And, while importing the server files, I have the option to copy the filebase to the title (or description or summary). I assume this isn't a problem for your script? (it will override were it is needed)

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Thu, 2010-11-11 17:49

I was able to add the pictures by chmodding everything with 777, using symbolink links and deferring the creation of thumbnails. One or two times the script aborted, so I just retried adding a smaller amount of albums.

After this I used your tool. It did not give any errors and finished by a "done!" statment. However, I do not notice any changes. Many albums are still called "album23" or something and the decription (or summary?) I gave those albums did not get imported. The description can be found in the XML file however. Here is an example of text I can find in the xml file, but did not get imported

Quote:
<row><f>32262</f><f>1</f><f>On the route from Bangkok to Chiang Mai this small town is known for the many temples one can visit here. One can also take a night-tour where the temples are wonderfully lit up. (Although it is very hard to take good pictures under these conditions, there are some nice pics of this)</f><f></f><f>28</f><f null="1" /><f></f><f>Ayuthaya</f><f>1156927139</f><f>1156927139</f></row>

Did something go wrong, or am I missing the point?

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Thu, 2010-11-11 21:59

My script does nothing to albums themselves, just to images.
It does filename and filesize comparisons can't exactly do that with a directory.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Thu, 2010-11-11 22:42

ah, ok. I understand.

BTW, when the script was running it outputted lines like

Quote:
updated item 103
updated item 106
updated item 109

The difference between two items ID's was always 3. Is this standard behaviour (nothing was skipped)?

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Thu, 2010-11-11 22:56

it only updates the item, the numbers in between are likely derivatives or albums...

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Thu, 2010-11-11 23:18

ah, I thought so. Just wanted to check.

In case you ever want to tweak your script, it would be nice if it could update the album info also. Although it doesn't have a size itself, perhaps the comparison could be based on the size of it's nested objects (or perhaps more easier, just only check for identical names and leave the size out of the equation? g2 auto-numbers albums ("album76"). changes are small that there are several albums with identical name).

In anyway, thanks a lot! Having to re-edit about 100 album title/descriptions beats editting 10.000 photo's! Now at least with some evenings of going through everything, I can still repair the damage (can't believe G2 had a stupid xml export that even the most knowledgable users did not get working)

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Thu, 2010-11-11 23:38

phpmyadmin 3.x can handle xml DB exports, but I've never actually tried. I believe php 5.3 is required for it too.

Glad my script could help.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
alecmyers

Joined: 2006-08-01
Posts: 4342
Posted: Mon, 2010-11-15 22:37
Quote:
can't believe G2 had a stupid xml export that even the most knowledgable users did not get working

The xml export/import was one of the very last "big" features added to G2 before development attention was switched to G3, so it didn't get the issues sorted out.

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Mon, 2010-11-15 23:11

ah, ok. Generally I find Gallery an impressive product (unbelievable how so many people deliver such great work). I can understand that with the many features this specific feature didn't get developped well enough. However, in such a case, I think it was better not to implement the feature at all, untill it was fully functioning. I didn't make a backup because the gallery upgrader suggested it would do it for me. If I knew it was buggy, I would have looked into it better before hand.

Anyway, otherwise it is a great platform (and ofcourse I should have made a full system backup in anyway. Lesson well learned ;-) )

BTW, do you think this feature still will be debugged in the future, or has all the development on G2 stopped since G3 is there?

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Mon, 2010-11-15 23:33

suprsidr, should I give the phpmyadmin a try (with your help/instructions)? I have myphpmyadmin 3.2.4. However, PHP is only version 5.2.14?

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Tue, 2010-11-16 00:34

for the 100 album titles?
make a backup of the current working db first.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Wed, 2010-11-17 08:40

yes, to retrieve the album titles and album description/summary (my personal description of each event/album is quite valuable to me).

I just now made a back-up from the G2 database according to the instructions on http://codex.gallery2.org/Gallery2:How_do_I_Make_Backups_of_My_Database (I had one extra field in myphpadmin, comments, that I left checked. See screenshot).

BTW, I moved (again) to a new server, installed G2.3.1 fresh and did an Add Server import. I have not run your script yet (ofcourse, I can do so if this is needed before proceeding).

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Wed, 2010-11-17 13:04
Quote:
BTW, I moved (again) to a new server, installed G2.3.1 fresh and did an Add Server import.

your db backup will not work w/ your new gallery. the correct procedure:
FAQ: How can I move my gallery installation from one server to another?

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Wed, 2010-11-17 13:43

I am confused. I cannot do all the steps since I did not make a backup when I needed to do it. To reiterate:

a) I had a Gallery 2.2 running on a windows machine. During a upgrade to 2.3 an automated backup XML was made. By accident, I deleted the database of the functioning 2.3 afterwards, only leaving me with a non-functioning G2.3 and an intact g2data folder.

b) I installed a fresh copy of G2.3 on a remote unix shared server. I FTP the intact g2data to the server and added the pictures succesfullly via the Add Server module.

c) 2.3 has now all the pictures in it, but not any meta information. I now made a backup of the g.2.3 via myphpadmin just for the security. (I thought you meant this with

Quote:
make a backup of the current working db first

.)

Do you mean, I should install G2.2 on the unix server (not G2.3), just FTP the original g2data to new server and not do an import?

My aim is to get the meta-information from the XML backup from 2.2 into the G2.3 . If I am complicating things by the fact that 2.2. was installed on my home (windows) server and 2.3 on a remote (unix) server, then I can ofcourse instead try the rescue operation on the home server by re-installing 2.2 fresh on the windows server.

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Wed, 2010-11-17 13:52

you should be working with the gallery you restored the other day. and move that.
Your gallery that was broken is gone.
Your item ids will never match up with your old xml backup again.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Wed, 2010-11-17 14:16
Quote:
you should be working with the gallery you restored the other day. and move that.

I was never able to fully restore it. I only achieved re-installing a fresh copy and doing an Add Server import. I thus now have a working 2.3 on the new server, including the imported pictures, but excluding stuff like album information, user accounts, etc.

Quote:
Your gallery that was broken is gone.
Your item ids will never match up with your old xml backup again.

does this mean my XML file is useless / has no use anymore (besides that your script could still retrieve picture titles etc)?

I thought that you meant with

Quote:
phpmyadmin 3.x can handle xml DB exports

there was a chance for me to import the meta information (album descriptions) from the XML backup, but may be I misunderstood you.

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Wed, 2010-11-17 14:31
 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Wed, 2010-11-17 14:32

you can still hand edit those ~100 album titles.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Fri, 2010-11-19 08:13

suprsidr, all right. I will have to give up on that and do the album info by hand. However, if I want to use your script for it's intented purpose to import titles of pictures, etc. I now get the following error when I run the script:

Quote:
Fatal error: Balloc() failed to allocate memory in /home/soepnl/public_html/gallery2/importXML.php on line 14

In responce the helpdesk altered the PHP settings to

Quote:
max_execution_time 1000
max_input_time 1000
memory_limit 712M

This did not however resolve the issue. After this the helpdesk increased the settings twice, without any effect. Finally, they concluded this script appearantly needs many GB of memory and hours of executing time, and obviously refused this for a shared hosting account.

How is this possible? I do have about 10.000 pictures in the database, but got this import tool working without any problem at my previous hosting company (which I believe were much more strict with their php settings; they did not allow php_exec).

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Fri, 2010-11-19 14:25

So now you've lost the previously restored version too?

I've updated the script above to do less actual interaction w/ gallery.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Sat, 2010-11-20 10:57

Yes. I moved to a new server, so thought I would re-do the steps freshly (install g2.3, import pictures and run your script). Could you attach the updated script? Then i will give it a try.

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sat, 2010-11-20 13:23
 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Sat, 2010-11-20 14:06

same result/error :-(

(I downloaded the zip, but the date mentioned in the php file was still 9/19/2010 so I hope I had the correct version)

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sat, 2010-11-20 16:45

that is the correct version, but it should be from 11/19 make sure you're overwriting the old one.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Sat, 2010-11-27 14:24

yes, I overwritten the original file with the updated version. Still get the same error:

Quote:
Fatal error: Balloc() failed to allocate memory in /home/soepnl/public_html/gallery2/importXML.php on line 14

The system admin recovered the following errors from the log file

Quote:
[18-Nov-2010 07:32:04] PHP Warning: simplexml_load_file() [<a href='function.simplexml-load-file'>function.simplexml-load-file</a>]: I/O warning : failed to load external entity &quot;/home/soepnl/g2/photos/galery_21945_2010_11_09_0006.xml&quot; in /home/soepnl/public_html/gallery2/importXML.php on line 14

[18-Nov-2010 07:32:04] PHP Fatal error: Call to a member function xpath() on a non-object in /home/soepnl/public_html/gallery2/importXML.php on line 16

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sat, 2010-11-27 15:12

my suggestion. when you've successfully recovered a "lost forever" gallery, don't throw it away.
I've never lost any of my galleries. I've never needed to restore from xml backup - but yet I spent my time to write a script to help others.
I don't get paid for this, and I feel you've already wasted plenty of my time.
That said.
The above error is obvious and in plain English but let me break it down:

Quote:
... function.simplexml-load-file ... failed to load external entity ... /home/soepnl/g2/photos/galery_21945_2010_11_09_0006.xml ... on line 14

make sure that path is correct and the file is readable.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
bobob

Joined: 2003-07-12
Posts: 110
Posted: Sat, 2010-11-27 17:06

sorry for asking. I double checked the path and permissions (perhaps the error retrieved from the log was from a prior incident). That is not causing the memory allocate problem. Thanks for your help anyway. Over and out.

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sat, 2010-11-27 18:05

try installing WAMP or similar on your local machine and import/repair your gallery then move it to a live server.
Otherwise I'm out of suggestions.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Thu, 2010-12-16 04:38