status of the gallery 1 -> 2 import tool?

Hey, Kim, glad to hear that you're interested! I am currently hard at work on the importation utility, and I could certainly use more people testing it. However, at the moment it only imports users and a skeleton structure of albums.

Woo hoo! I have successfully imported albums and images into my development G2 installation. However...
Importing non-images hasn't been tested
International text hasn't been properly tested
A lot of metadata is being left behind
Most errors are NOT being trapped and handled. This includes trying to import an album into a place where an album of the same name already exists. It just dumps you into a cold hard error page.
Album ownership and permissions are not being imported AT ALL
Bugs abound!

That still leaves room for people to play with the migration module and make requests so I know what functionality other people want.

What can you do to help?
Run the unit tests. Tell me if anything comes up red.
Make a copy of your gallery's (version1) albums. Try to import the copy. Don't freak out if there are errors (there will be), but let me know what you run into. This isn't critical right now since I have plenty fix already, but it is nice to know that people are interested, and alert users are very appreciated.

I get the following error with the latest version that i did get from cvs

Error (ERROR_BAD_PARAMETER): Missing or not readable file: /www/test-familie-berg.org/albums/.users/8
in file modules/migrate/classes/Gallery1DataParser.class at line 66 (gallerystatus::error)
in file modules/migrate/classes/Gallery1DataParser.class at line 315 (gallery1dataparser::loadfile)
in file modules/migrate/ConfirmImport.inc at line 653 (gallery1dataparser::getuserfieldsbyuid)
in file modules/core/SiteAdmin.inc at line 94 (confirmimportview::loadtemplate)
in file modules/core/classes/GalleryView.class at line 205 (siteadminview::loadtemplate)
in file main.php at line 259 (siteadminview::doloadtemplate)
in file main.php at line 24

Hello jmullan,

Can you make this wordaround with a option so users can test this.

Hi !
I tried today's release (11.27 2003), tried to migrate my albums, and ...
- it imported my users successfully, as far as I can tell
- it started to import my albums, but after a while, it stopped. no error, but it stopped, and only part of my albums were migrated.
- I have some albums that only contains sub albums and no pix. There albums were imported correctly, but don't have any thumbnail set.
- I haven't tried it today, but last time I tried to import albums which contain titles with accents (é è ...), it didn't import them correctly.

OK, I think that's all I saw, I hope it helps to see which problems remain.

good luck guys !

sam

My earlier problem was caused by very high load on our system. I have now tried the migration module again, and have discovered theese issues (same CVS as before):

Try to import all albums, I usually get this:

So then I descided to try each top album by itself. That worked pretty well, except two occurences. I traced it down to theese two albums (supplied with the errorcode it generated):

The last problem, I'm pretty sure is cause migrate doesnt do an check for if an album is empty (no items in the album)

Hope that helps to some extent...[/b]

nice to hear it helped!

One thing still missing is the import of the highlight from nested albums, correct?

give me a reply or something when u you have some CVS updated, and I'll try it out...

One idea/suggestion:
split the migration up in parts, so the script doesnt time out (for big galleries). I've seen a nice solution to this with 4images, where they have a javascript to automatic continue the script.

cheers,
hawkin.

well, despite my implied boundless energy, I'm afraid that my sore throat and accompanying wooziness is getting the best of me. tomorrow night, I swear.

I was at least listening to Barry White.

I have get the cvs from today and did try the import tools and itś working, only when i select two or more albums i get the following error.

Error (ERROR_BAD_PARAMETER): Missing or not readable file: /www/test-familie-berg.org/albums/.users/1032082730:1755202938

* in file modules/migrate/classes/Gallery1DataParser.class at line 66 (gallerystatus::error)
* in file modules/migrate/classes/Gallery1DataParser.class at line 331 (gallery1dataparser::loadfile)
* in file modules/migrate/ConfirmImport.inc at line 676 (gallery1dataparser::getuserfieldsbyuid)
* in file modules/core/SiteAdmin.inc at line 94 (confirmimportview::loadtemplate)
* in file modules/core/classes/GalleryView.class at line 205 (siteadminview::loadtemplate)
* in file main.php at line 259 (siteadminview::doloadtemplate)
* in file main.php at line 24

I got that error on a G2 build from last week but todays (12/14) cut worked w/o error.

This was my 1st successful attempt at doing the import. (only moved about a dozen albums). I'm on 1.4.1 RC3/php-nuke 6.9/Windows NT

If you want, you can check out the before and after:
G1: http://www.fedak.net
G2: http://www.fedak.net/modules/gallery2/main.php

Things I noticed:
- Album heirarchy is lost during import
- The matrix layout doesn't wrap captions so that the pictures stay in an evenly spaced grid
- Album header text seems to get truncated
- Some picture captions seem to get truncated
- Album dates not transferred

Very impressed with G2 thus far,
-fedak

Highlight from a highlighted nested album now works.

Sub-albums as highlights

Fixed...

Empty albums

Fixed...

Resetting G2 data

I will give this some consideration.

International text

I'm not 100% sure about international text. G1 wasn't as firm about specifying a character set, so G1 metadata could be in any character set. G2 is UTF-8 (at my recommendation), so some stuff may or may not work right. I could simply translate any extended characters directly into Unicode html entitities, but that won't work for multi-byte characters. This is a non-trivial issue, and I want to either handle it perfectly or put the onus completely on the user to fix their data to be UTF-8 - I don't want to support only French or only Japanese.

Nuke users
I need to install a nuke of some sort and build a gallery as a module under it so I can have a good hunk of test data to futz with this.

Album ownership and permissions
Album ownership should now be correctly imported (if the owner exists or is imported first). Permissions will involve more work, but they will be possible.

Importing non-images
I tested the importation of an mpg, that worked fine. I don't know about anything else, I haven't tried.

Archive G1 data
I will be adding the ability to store the G1 data in a new field or table, so that data will be available in the future- like if we implement custom fields after everyone migrates their galleries. This is a fairly critical item.

Figure out uid weirdness
At one point, all uids in a G1 installation were 12345678:12345678, or 12345678;12345678. Recently we changed the spec to 12345678_12345678 for everyone. Hopefully every uid and the uid files should all be in the same format, but I should make sure that we can always get the user data, even if the uid is specified in one format and the file is named in the other format.

There is other weirdness where some people's user data is being listed as non-existant or unreadable. I'm not sure what is going on, and it may be a while before I get that figured out. Or maybe that will be next.

layout-related things
Unfortunately, Matrix doesn't seem to allow for things like per-album background colors. I'm not sure what to do here, but I will talk with bharat to see if this data can be saved for future use.

custom fields
This is something that I will have to talk to bharat about - if this isn't implemented in G2 I won't be able to import the data.

A lot of metadata is being left behind
There is other miscellaneous data which I should import.

Album hierarchy should not be lost unless something is really broken. This person had a lot of albums (like me), so maybe they weren't being properly parsed.

This is a matrix problem, I hope. ;)

It shouldn't be truncated. Can you double check that it isn't simply Matrix cutting off titles? Maybe when you view the album or picture the full text will show up.

I've been debating with myself about importing album modification dates. I'm pretty sure that migrating from G1 to G2 should be considered an album modification. ;)

However, I might not be picking up creation dates. I will double-check that. Click counts should be correctly imported, along with the date since we started counting clicks for that particular item. I need to doublecheck this, since I might be importing for albums only and not images, or vice versa.

Timeout issues
Since I am one of those folks with a large gallery, this is important to me, too. Right now this module has still not been optimized. I can't import my big gallery in one fell swoop. Rest assured that I will make this whole thing faster - but there might be simple limits to how fast this will work, and how much data can be imported at once.

This one is somewhat new to me - could you try this album again and let me know if you still get the error?

A HUGE thanks to everyone testing so far - it really helps to keep me focused, and to help me find those things that I might otherwise miss.

Let me address these points since they aren't migration related (so they aren't necessarily things that Jesse is familiar with).

In G2, the summary is limited to 256 characters. It's a summary, not a description so it's probably going to remain that way. We may make the text box larger in the GUI for ease of use (there's still a lot of GUI polishing left to do!) but the goal in G2 is to get finer grained control over the data which is why we've split the concept of a "caption" out into title, summary and description. The description field is virtually open ended (depends on the database, but most dbs can handle 64K of description data).

This is for security reasons. The current plan is to allow you to enter BBCode (similar to what we use here in the forums). This will give you a reasonable level of HTML beautification without opening the door for any nasty cross-site-scripting security issues that plague apps like this. We'll probably add a global setting to allow you to enable raw HTML, eventually.

That's controlled by the layout (and we have the concept of interchangeable layouts) so it's something that will be easily changed in the future. Again, unpolished. Give that a little time.

Yep, this is a known issue on my radar. I'll fix it pretty soon.

More unpolished Matrix layout issues -- this will get fixed in about a month (we have to do some other stuff first).

G2 creates the images on demand then caches them. So the first time you view a thumbnail you have to wait for it to be created. Subsequent page reloads should be significantly faster. It'll still be a little slower than G1, but it'll maintain that same speed as you get up to 100K images (where G1 performance will drop off dramatically over about 15-20K images). Also, I still haven't done any optimization -- I expect a 1.5-2x speedup before we get to alpha.

Right .. I actually don't know where to take you if you cancel at that point. There's no general purpose landing page if you cancel from that point since you can get there from different places. Any thoughts on that?

-Bharat

fedak: could you make sure that you have upgraded absolutely every album in G1?

Also, due to the large number of albums that you are importing, I think that you might be out of luck until a major optimization of the migration module (which will happen). I'm in the same boat.

As to loss of caption data... I'm not sure what the best approach to take would be. I have a lot of long captions myself, so again, I'm in the same boat.

I would suggest that you take general G2 comments into a new thread. Go nuts! We love the feedback. It is cluttering up this thread a little, but I will live. ;)

Yea .. that's been in the back of my mind for a while. It's definitely possible (there's a framework for doing this which is how the login link always returns you to the same page after you've logged in) but I'm going to let it simmer for a while longer and see if a more elegant answer naturally emerges.

Yep, there's a "v2.0-pre-alpha" group in the bug tracker on the SF.net project page -- log your bugs with that group and I'll take care of them. Thanks!

Yep, it should work on Windows but I haven't tried it lately so there are probably a few little things that I've changed recently that have broken windows support. File a bug about that, would you please? I'll fix it when I get a chance.

Don't worry about the unix path separators or even the weird intermingling of unix and windows path separators when you're using windows -- PHP is smart enough to do the right thing with those. Thanks for the report, though!

Yeah, I'm on 1.4.1 (and have incrementally been on every released version for the past ~2 years, so it's quite possible that I've aquired some baggage). Its quite likely that I've not visited all of my albums since my 1.4.1 upgrade.

Is there a quick way validating the integrity of my G1 album heirarchy? Going to somewhat suck having to manually navigate to 200+ albums. Seems like some form of preflight validation on the G1 structure would save alot of future headaches.

I'll PM you with a read-only ftp to my G1 directory. Can get you a zip of my .dats when I get home this evening if you'd like that as well.
-fedak

Shell access, yes. Ksh no (it's a windows box).

Send me the script and I'll see what I can do. I should be able to port it over to a bat file (or worst case install a windows version of ksh)

-fedak

Sonofa!

Hrm. I may end up having to install G2 on a windows box to work on it.

That's a G1 issue, and I filed a bug report and partial patch. I need to chat with the G1 leads to get the update worked into the upgrade_albums process, since all albums will have to be fixed. This is a terrific catch, and will probably fix a few sneaky bugs that we may or may not have been previously catching.

Of course, it does mean that in order to migrate one's G1 gallery, one will probably have to upgrade to 1.4.2 first. That isn't necessarily a bad thing, but it might be irritating to some users.

I still have the upgrade problem rolling around the back of my head - how to upgrade all albums without using the recursive tool on a windows server. Do you have cygwin or any unix tools available? Unix find and wget could probably do the work, even in a windows environment (while running natively from the command line).

You could conceivably download your AlbumDB.dat file and hand edit it into a .bat file to request every album via wget. You could install the php windows binary and custom code something to read a locally saved AlbumDB.dat file and request each album in turn.

That might be a fun php-gtk project. Hmm. I've been meaning to futz with that.

I really don't know how I convinced myself to make that cvs commit without running through the validation process. The latest build should be fixed.

Exactly. The tool inside gallery is recursive, I think (but it might not be, because it doesn't have to be).

I don't know your level of experience (but it does seem high). A lot of users want to be given the tools to do things, some want to write the tool themselves but need a push, and some just figure things out on their own. Some people are willing to go out and get whatever tools they need, and some want to use what they know already. I'm just trying to provide enough context that you can decide what you want to do. I have a ksh script written already, but that's it.

Whatever you use to request each url does not need to be logged in to upgrade individual albums. Any request of an album will upgrade it (unless there is a problem).

fedak=awesome!

Excellent.

I'm aware of that. I'm hoping to get a change into G1 so that I don't have to use the very expensive G1 process to determine album order. For the moment you will just have to suffer. ;)

Excellent. Did you write a script? I put together a php-gtk app to do the same thing last night, and then tested it on your site. I think that I will make a php-cli version too, then package those and the ksh script together to provide some easy options for users of large galleries. It is critical that all albums are updated so that uids are up-to-date everywhere in G1.

That's a new one. I will look over that code. Thanks!

No, I have them, but haven't done anything with them as of yet. I think that I need to use bharat's album data spider tool to validate everything. Or maybe it's just the $album->fields['name'] thing, which is a sneaky but critical bug (at least as far as the migration module is concerned).

I think that may have something to do with the number of albums that your main gallery has. Does this happen when you try to import just a handful of albums, or only when you do a large mass? I need to review the locking mechanisms too, because I'm really not that familiar with how they work.

Since bharat just set up the BBCode thing, I will likely be converting html to BBCode. This will probably be user-selectable. I will be doing similar things to try to convert extended characters into UTF-8, but I don't know how wise that process will be.

Also, at some point I will be adding a piece to the database to store all of the old album and image metadata, so that we could conceivably parse it at a later date when additional features are implemented.

I've got a long long task list. Thanks again for all your testing work!

Ok, found and resolved the issues that were causing my incomplete album list and missing title problems:

There were case issues with some of the parentAlbumName fields in my album.dat files. (i.e. the strings were correct, but with incorrect cases.)
Since the migration module builds it's tree by using the backwards (subalbum->album) link, it was unable to attach these subalbums to the tree. (An error message in getAlbumHierarchy for when isset($tempAlbums[$parentName]) returns false would really help matters here)

I'm not sure exactly when/how this happened, but I've had case issues with G1 at various points along the way- usually due to mod_rewrite doing ugly things to my urls. Since the windows filesystem isn't case sensitive, I've just changed the G1 lookups to strcasecmps to get it working. Looks like this just came back to bite me.

Wrote a little php routine to do a top down traverse of the G1 tree and set the parentAlbumName from the parent album's name. If anyone needs it, it can be found here: http://www.fedak.net/modules/gallery/fix_album_parents.txt. (Use at your own risk, this is the first thing I've ever written in php )

-fedak

Back from vacation and had some time to play with the latest dev build. Things are looking much better, and I was able to migrate some albums again.

G1: http://www.fedak.net/
G2: http://www.fedak.net/modules/gallery2/main.php

- There's two places in Gallery1DataParser.class where a path containing a hard coded '/' gets passed to loadAlbumFields. This causes the import to fail on windows. (The album names come back with '/' appended to them). Changing them to '\\' fixes the issue on windows. Code needs to be modified to use $slash.

- With the above issue fixed, I was able to successfully import albums (the ERROR_LOCK_TIMEOUT from before the holidays appears to be fixed). This error may have been caused by me not having netpbm/imagemagick configured after a reinit- the migrate module seems to require these modules but does not check for their activiation.

- My "YosemiteWeekend" album is being migrated w/o a highlight thumbnail. This album contains only subalbums.

- Getting the following error migrating my "SonoraStanislaus" album tree. I'll look into this over the next day or two and make sure it isn't a data issue with my album files.

Error (ERROR_COLLISION):
in file c:\nuke\html\modules\gallery2\modules\core\classes\GalleryAlbumItem.class at line 203 (gallerystatus::error)
in file c:\nuke\html\modules\gallery2\modules\core\classes\GalleryAlbumItem.class at line 122 (galleryalbumitem::_createdir)
in file c:\nuke\html\modules\gallery2\modules\migrate\ConfirmImport.inc at line 272 (galleryalbumitem::create)
in file c:\nuke\html\modules\gallery2\main.php at line 146 (confirmimportcontroller::handlerequest)
in file c:\nuke\html\modules\gallery2\main.php at line 24

- Custom thumbnails (i.e. ones generated using the G1 java thumbnail editor) are not preserved during migration

-fedak

The following list is in the source of ConfirmImport, and is my checklist for metadata.

LEGEND:
+ done (or something approximating done)
- will not do

/* album object
			     * Array of example data
			     * (
			     *   +[title] => Test Album One
			     *   +[description] => No description
			     *   +[summary] =>
			     *   -[nextname] => aaa
			     *    [bgcolor] =>
			     *    [textcolor] =>
			     *    [linkcolor] =>
			     *    [font] => verdana
			     *    [border] => 1
			     *    [bordercolor] => black
			     *    [returnto] => yes
			     *   +[thumb_size] => 150
			     *   +[resize_size] => 400
			     *   +[rows] => 5
			     *   +[cols] => 5
			     *    [fit_to_window] => no
			     *    [use_fullOnly] => no
			     *    [print_photos] => none
			     *    [use_exif] => yes
			     *    [perms] => Array
			     *        ( [canRead] => Array
			     *            ( [everybody] => 1 )
			     *            [canViewFullImages] => Array
			     *            ( [everybody] => 1 )
			     *        )
			     *   +[parentAlbumName] => .root
			     *   +[clicks] => 0
			     *   +[clicks_date] => 1056014708
			     *    [display_clicks] => yes
			     *    [public_comments] => yes
			     *   -[serial_number] => 8
			     *    [extra_fields] => Array
			     *        ( [0] => Description
			     *        )
			     *   -[cached_photo_count] => 2
			     *   -[photos_separate] => 1
			     *   +[name] => album01
			     *   +[owner] => 1054014347u1233740678
			     *   -[last_mod_time] => 1056015246
			     */
/* albumitem Object
			     * (
			     *   +[image] => image Object
			     *        (
			     *           +[name] => canon_ae_1
			     *           +[type] => jpg
			     *           -[width] => 225
			     *           -[height] => 165
			     *           -[resizedName] => {
			     *                [thumb_x] =>
			     *                [thumb_y] =>
			     *                [thumb_width] =>
			     *                [thumb_height] =>
			     *                [raw_width] => 225
			     *                [raw_height] => 165
			     *               -[version] => 10
			     *            )
			     *           -[thumbnail] => image Object (
			     *               -[name] => canon_ae_1.thumb
			     *               -[type] => jpg
			     *               -[width] => 150
			     *               -[height] => 110
			     *               -[resizedName] =>
			     *               -[thumb_x] =>
			     *               -[thumb_y] =>
			     *               -[thumb_width] =>
			     *               -[thumb_height] =>
			     *               -[raw_width] => 150
			     *               -[raw_height] => 110
			     *               -[version] => 10
			     *            )
			     *
			     *           -[caption] => canon_ae_1
			     *            [hidden] =>
			     *           +[highlight] => 1
			     *           -[highlightImage] => image Object (
			     *               -[name] => canon_ae_1.highlight
			     *               -[type] => jpg
			     *               -[width] => 200
			     *               -[height] => 147
			     *               -[resizedName] =>
			     *               -[thumb_x] =>
			     *               -[thumb_y] =>
			     *               -[thumb_width] =>
			     *               -[thumb_height] =>
			     *               -[raw_width] => 200
			     *               -[raw_height] => 147
			     *               -[version] => 10
			     *            )
                             *           +[isAlbumName] =>
			     *           +[clicks] =>
			     *           +[keywords] =>
			     *            [comments] =>
			     *            [uploadDate] => 1056014783
			     *           +[itemCaptureDate] => Array (
			     *               -[seconds] => 23
			     *               -[minutes] => 26
			     *               -[hours] => 04
			     *               -[mday] => 19
			     *               -[wday] => 4
			     *               -[mon] => 06
			     *               -[year] => 2003
			     *               -[yday] => 169
			     *               -[weekday] => Thursday
			     *               -[month] => June
			     *               +[0] => 1056014783
			     *            )
			     *
			     *           -[exifData] => Array (
			     *               -[File name] => canon_ae_1.jpg
			     *               -[File size] => 8070 bytes
			     *               -[File date] => 2003:06:19 04:26:23
			     *               -[Resolution] => 225 x 165
			     *               -[Jpeg process] => Baseline
			     *            )
			     *
			     *            [extraFields] => Array (
			     *            )
			     *           -[version] => 10
			     *)
			     */

I must admit that I have been very distracted by the holidays.

PHP itself doesn't care, the issue here seems to be with string manipulation of album paths rather than with using the php functions for file access.

My preference for this would be some user settable control of the field mapping. Specifically, I'd like to map the G1 item captions -> the G2 item description, G1 filename -> G2 item title, G1 album summary -> G2 album description.

-fedak