G3 Beta 1 - Import from Gallery 2.3 stuck

I think this log is more relevant.

What does "Missing object for 41626" means? What should I do?

2009-06-14 11:32:30 +02:00 --- error: Gallery2 call failed with: Error (ERROR_MISSING_OBJECT) : Missing object for 41626 <b>in</b> modules/core/classes/GalleryStorage/GalleryStorageExtras.class <b>at line</b> 2054 (GalleryCoreApi::error)
<b>in</b> modules/core/classes/GalleryStorage/GalleryStorageExtras.class <b>at line</b> 98 (GalleryStorageExtras::_identifyEntities)
<b>in</b> modules/core/classes/GalleryStorage.class <b>at line</b> 313 (GalleryStorageExtras::loadEntities)
<b>in</b> modules/core/classes/helpers/GalleryEntityHelper_simple.class <b>at line</b> 71 (GalleryStorage::loadEntities)
<b>in</b> modules/core/classes/GalleryCoreApi.class <b>at line</b> 2361 (GalleryEntityHelper_simple::loadEntitiesById)
<b>in</b> modules/core/classes/GalleryChildEntity.class <b>at line</b> 145 (GalleryCoreApi::loadEntitiesById)
<b>in</b> modules/core/classes/GalleryFileSystemEntity.class <b>at line</b> 283 (GalleryChildEntity::fetchParent)
<b>in</b> modules/core/classes/GalleryDataItem.class <b>at line</b> 377 (GalleryFileSystemEntity::fetchContainerPath)
<b>in</b> /var/www/www.serialphotographer.com/gallery3/modules/g2_import/helpers/g2_import.php <b>at line</b> 385 (GalleryDataItem::fetchPath)
<b>in</b> /var/www/www.serialphotographer.com/gallery3/modules/g2_import/helpers/g2_import_task.php <b>at line</b> 126 (g2_import_Core::import_item)
<b>in</b> ??? <b>at line</b> 0 (g2_import_task_Core::import)
<b>in</b> /var/www/www.serialphotographer.com/gallery3/modules/gallery/helpers/task.php <b>at line</b> 78
<b>in</b> /var/www/www.serialphotographer.com/gallery3/modules/gallery/controllers/admin_maintenance.php <b>at line</b> 140 (task_Core::run)
<b>in</b> ??? <b>at line</b> 0 (Admin_Maintenance_Controller::run)
<b>in</b> /var/www/www.serialphotographer.com/gallery3/modules/gallery/controllers/admin.php <b>at line</b> 50
<b>in</b> ??? <b>at line</b> 0 (Admin_Controller::__call)
<b>in</b> /var/www/www.serialphotographer.com/gallery3/system/core/Kohana.php <b>at line</b> 291 (ReflectionMethod::invokeArgs)
<b>in</b> ??? <b>at line</b> 0 (Kohana::instance)
<b>in</b> /var/www/www.serialphotographer.com/gallery3/system/core/Event.php <b>at line</b> 209
<b>in</b> /var/www/www.serialphotographer.com/gallery3/system/core/Bootstrap.php <b>at line</b> 55 (Event::run)
<b>in</b> /var/www/www.serialphotographer.com/gallery3/index.php <b>at line</b> 72

Yup, I'm slowly adding more graceful degradation code to deal with corruption. I believe that the latest code will deal with this failure gracefully.
---
Problems? Check gallery3/var/logs before you post! and file bugs here!
Latest code/upgrading: http://codex.gallery2.org/Gallery3:Upgrading
Latest git: http://codex.gallery2.org/Gallery:Using_Git

Using the current code from git my import from G2 is still hanging after 3721 items because of some missing item. Here's the log:

2009-06-16 17:25:29 -04:00 --- error: Gallery2 call failed with: Error (ERROR_MISSING_OBJECT) : Missing object for 11458 in modules/core/classes/GalleryStorage/GalleryStorageExtras.class at line 2054 (GalleryCoreApi::error)
in modules/core/classes/GalleryStorage/GalleryStorageExtras.class at line 98 (GalleryStorageExtras::_identifyEntities)
in modules/core/classes/GalleryStorage.class at line 313 (GalleryStorageExtras::loadEntities)
in modules/core/classes/helpers/GalleryEntityHelper_simple.class at line 71 (GalleryStorage::loadEntities)
in modules/core/classes/GalleryCoreApi.class at line 2361 (GalleryEntityHelper_simple::loadEntitiesById)
in modules/core/classes/GalleryChildEntity.class at line 145 (GalleryCoreApi::loadEntitiesById)
in modules/core/classes/GalleryFileSystemEntity.class at line 283 (GalleryChildEntity::fetchParent)
in modules/core/classes/GalleryDataItem.class at line 377 (GalleryFileSystemEntity::fetchContainerPath)
in /usr/local/gallery3/modules/g2_import/helpers/g2_import.php at line 404 (GalleryDataItem::fetchPath)
in /usr/local/gallery3/modules/g2_import/helpers/g2_import_task.php at line 126 (g2_import_Core::import_item)
in ??? at line 0 (g2_import_task_Core::import)
in /usr/local/gallery3/modules/gallery/helpers/task.php at line 78
in /usr/local/gallery3/modules/gallery/controllers/admin_maintenance.php at line 140 (task_Core::run)
in ??? at line 0 (Admin_Maintenance_Controller::run)
in /usr/local/gallery3/modules/gallery/controllers/admin.php at line 50
in ??? at line 0 (Admin_Controller::__call)
in /usr/local/gallery3/system/core/Kohana.php at line 291 (ReflectionMethod::invokeArgs)
in ??? at line 0 (Kohana::instance)
in /usr/local/gallery3/system/core/Event.php at line 209
in /usr/local/gallery3/system/core/Bootstrap.php at line 55 (Event::run)
in /usr/local/gallery3/index.php at line 82

2009-06-16 17:25:29 -04:00 --- error: @todo G2_FUNCTION_FAILED in /usr/local/gallery3/modules/g2_import/helpers/g2_import.php at line 794:
#0 /usr/local/gallery3/modules/g2_import/helpers/g2_import.php(404): g2(Array)
#1 /usr/local/gallery3/modules/g2_import/helpers/g2_import_task.php(126): g2_import_Core::import_item(Array)
#2 [internal function]: g2_import_task_Core::import(Object(Task_Model))
#3 /usr/local/gallery3/modules/gallery/helpers/task.php(78): call_user_func_array('g2_import_task:...', Array)
#4 /usr/local/gallery3/modules/gallery/controllers/admin_maintenance.php(140): task_Core::run('1')
#5 [internal function]: Admin_Maintenance_Controller->run('1')
#6 /usr/local/gallery3/modules/gallery/controllers/admin.php(50): call_user_func_array(Array, Array)
#7 [internal function]: Admin_Controller->__call('maintenance', Array)
#8 /usr/local/gallery3/system/core/Kohana.php(291): ReflectionMethod->invokeArgs(Object(Admin_Controller), Array)
#9 [internal function]: Kohana::instance()
#10 /usr/local/gallery3/system/core/Event.php(209): call_user_func(Array)
#11 /usr/local/gallery3/system/core/Bootstrap.php(55): Event::run('system.execute')
#12 /usr/local/gallery3/index.php(82): require('/usr/local/gall...')
#13 {main}
2009-06-16 17:25:29 -04:00 --- error: Uncaught Exception: @todo G2_FUNCTION_FAILED in file modules/g2_import/helpers/g2_import.php on line 794

So how do I fix item 11458?

Ok, I found rows in ChildEntity that had the parentID of 11458, so I deleted those. I went through a few of the other tables looking for 11458 and didn't find any, so I started the import over again. It hung at the same place with the same error and 11458 missing object.

Any other tables I should be looking in to clean this out?

Thanks for the tip ... I had never tried doing a full-text search like that.

Anyway, the search came back negative for the ID, so I'm still stuck.

You're right. I forgot to mention I reset Gallery cache via the web interface, I ran the "optimize database" too. I have also deleted the g2data/cache folder on disk just to be sure. Keep trying

Thanks bharat! I've pulled the latest code via git and have just kicked off another import. I'll let you know how it goes ...

Ah that makes sense. We don't support BMP images in Gallery 3. We should probably scan your Gallery 2 for all the edge cases and report to you before the import starts...

---
Problems: Check gallery3/var/logs first!
file a bug or feature request | upgrade to the latest code | use git

You don't have any log files under /var/logs?

Geoff, get the latest code using git and you'll find that it works...

Maybe try turning on extra logging, like described here?

http://gallery.menalto.com/gallery_3.0_beta_1_released?page=1#comment-310536

Have you disabled any PHP accelerators?

@Geoff, the import has to be on the same machine as your G2. What version of G2 are you importing from? Can you package up your var/modules/g2_import dir and attach it here so that we can examine it?
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git | help! vote!

Weird. I'm guessing that there's some unexpected issue in upgrading from Gallery 2.1 (I've only tested with Gallery 2.2 and 2.3). Can I get ssh access to your box to tinker around with this?
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git | help! vote!

I'm trying to import about 29,000 images (and I didn't even add any since Dec 2008) and it just hangs....

Says importing album 183 of 183 and it's at the very beginning for the progress and just sits there. I left it run for about 20 mins with nothing.

I then created the local.php file. Anything else I need to do to troubleshoot?

@Ranger187: are you using the latest code? Did you add that log line yourself? I don't see it in the current code. If you can reproduce this with the latest code I'm happy to dig into it.
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git | help! vote!

@Bharat, I had the issue with it not working when it was setup. ie: http://// and you told me to pull the latest. I did. Then when I try to import my already massive gallery, it shows that error in the /var/logs. All I added was the local.php to the root. Nothing more.

I'm probably one of few on here with a gallery this massive and even re-creating the thumbnails in 2.3 always failed/timed out.

What would really be great, is if during the import, you could SELECT folders (a few, or just 1 at a time) instead of pulling 29,000+ images at once.

What do I need to do/add so that while it "says" it's importing, but the progress bar does nothing but sit for days I can physically see what it's doing? If I have to edit something, or add something, let me know! I'll do it. But for now, the import system to me is not informative enough.

Only change made was added "local.php" to root with code:

<?php defined("SYSPATH") or die("No direct script access.");
define("IN_PRODUCTION", false);

// Set the error reporting level. Use E_ALL unless you have a special need.
error_reporting(E_ALL);

// Disabling display_errors will effectively disable Kohana error display
// and logging. You can turn off Kohana errors in application/config/config.php
ini_set('display_errors', true);

Now the stats again:
Platform Information .Operating System: WINNT
Apache: Unknown <--- IIS7 (Windows Server 2008 x64 SP2)
PHP: 5.2.8
MySQL: 5.1.35-community
Server load: Unavailable

Gallery Stats .Version: 3.0 pre beta 2 (git)
Albums: 182
Photos: 5 <--- FAIL!!

You tell me what ya want me to do and I'll do it. Thanks bro.

Sorry for the delay. You're using Windows, which is unsupported, but I'm going to assume that is not the problem (I hope it's not, else I'm wasting my time here). So. The first thing we need to do is, get the latest code (see my sig below) and try the import again and figure out what the latest error message is in your logs. The one you posted before must have been generated by an older version of the code because I can't find it now. Also, after it hangs you can browse to Admin > Maintenance and click "[view log]" on your import task and maybe we can find more clues there.
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git | help! vote!

I've imported 15k photos and it worked properly, so I'm guessing it's not the size it's the diversity and possibly some mild corruption in your G2 data. We'll sort it out.. but let's start from the beginning. Let me know what error you get using a clean install of the latest git code and we'll start narrowing it down.
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git | help! vote!

There is no admin > maintenance > view log for the import........ This is what I have so far.....

Last Updated Name Status Info Owner Action
2009-Jul-13 01:18:39 Import from Gallery 2 0% Complete Importing albums (9 of 183) admin cancel

It's importing now. But at this rate, it will take a month. It's on 2000 out of 32,000+. I'm going to setup a new vmware session on the other box which is 5x faster and see if it speeds up.

Yeah, I have to say with people with large databases, this won't be a good solution and will be a big determing factor in using the product from G2 > G3.

I'm half tempted to say screw importing, just copy/paste the images from site 1, to new site and import.

Well, I got about 17,000 imported, then the log file just fills up. Bharat, should I email the log to you? It's about 6 megs.

That seemed to work.