backup and/or export

RwD
RwD's picture

Joined: 2005-01-09
Posts: 383
Posted: Tue, 2005-01-18 11:40

Will there eventually be a backup feature for the gallery?
And also, an export feature that can export albums, backups or basically anything that is exportable

I tried the search, but I didn't read anything conclusive...

 
bharat
bharat's picture

Joined: 2002-05-21
Posts: 7994
Posted: Wed, 2005-01-19 02:12

I know that somebody was working on a backup/export module, but I don't know what the status is. It's somewhere in this forum though.

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Wed, 2005-01-19 19:10

it's here
I didn't abandon it... But I am rewriting it - significantly.

Stay tune :wink:

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Sun, 2005-01-23 05:26

I rewrote it drastically... Here is the backup portion. While I realize that backup without restore is not very useful - I want to get started with the code review.

I hope I took all Bharat's comments from the first review into consideration (probably, more likely 30-40% 8) but I tried). Anyway, restore is coming soon, too

Any comments on functionality are welcome!

 
Einstein
Einstein's picture

Joined: 2003-10-13
Posts: 105
Posted: Sun, 2005-01-23 12:13

Backup, .. I assume this is purely manual?

I can contribute with an alternative solution. Last week I made an PHP scripts that automaticly do increamental backups to my homecomputer/server on regular basis through rsync. It's really practical when the bandwidth used is minimal and you can keep the history of changes.

Anyway, I have to clean it up a bit before it can be gallery compliant. It's more like a tutorial then a module. Could be nice if the restore process could be used through this module.

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Sun, 2005-01-23 19:46

Einstein, I am not sure what you mean by 'Backup, .. I assume this is purely manual?'

Computer does it - so it qualifies for automatic, eh? Seriously, though, it allows to have application-level backup of your albums and album trees making it schema- and database- independent.

I am using it for three reasons... First, I have a gallery on ISP server with limited space (it is 3GB but it is still limited). I have several authors that put their pictures on my site - I am just the sysadmin. So, I backup less popular or older albums, host it at my home computer (which is also accessible from the web, but it is slower, I may turn it off if I so decide, etc.) and delete those albums from the ISP host. With G1 the only backup that is available is full backup of all albums.

Second, I have a smaller host (500MB or so) that I use for G2 testing. I migrated G1->G2 on my home computer (in pieces) and then by using backup/restore I can move these migrated albums from home comp to test server. I don't know how else could I migrate the albums from large production G1 box to small dev G2 box.

Third, I made some tweaking with couple albums. I did most of that stuff on the plane, on my laptop. Once I land - I backed up the album from the laptop, and restored it to the "real" box.

So, these use cases may not be that common but this is what I wrote the backup for. I believe, your backup scripts is more typical backup procedure - backup up; and restore in case something goes wrong. Somebody suggested that my module is better called export/import. He is porbably right...

Hope, that explains the difference (and hope, I understand the difference between my module and yours correctly).

 
Einstein
Einstein's picture

Joined: 2003-10-13
Posts: 105
Posted: Sun, 2005-01-23 20:54

Thank you for a good explantion of how it works.

What I mean with manual is that you are triggering the process by clicking somewhere. Like you click Export and so on.

What I mean by increamental backup is that you set up a live system and backup system. And if your live system get deleted, hacked or screwed up somehow, you restore your copy from your backup system. Even if your system get screwed up for a week ago you can retrieve the exact version for week back.

And by "automatic" i mean a system where the backup is run on reglular basis from a cronjob. Or it can be integrated into Gallery to run as some background process.

To read more about the method I currently is using:
http://www.jdmz.net/ssh/
http://www.enterprisenetworkingplanet.com/netos/article.php/10951_1573881_2

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Mon, 2005-01-24 03:05

Oh, I understand now. In this sense, my backup is definitely manual. As I said, this is less of a backup system (where you need the backup only in case "your ... system get deleted, hacked or screwed up somehow"), but rather the export facility where you export from one system and import into another.

Obviously, for there are other criteria for backup (in your definition) module; including automation, incremental, and scheduling. But I would consider database backup for this purposes; rather than application. Granted - it's a completely different topic 8)

 
mindless
mindless's picture

Joined: 2004-01-04
Posts: 8601
Posted: Mon, 2005-01-24 23:54

virshu, i haven't tried this code yet, but curious if you can export a single album (and all nested albums, presumably) and import it into another G2?

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Tue, 2005-01-25 03:16

exactly! that's the purpose of this module. Import is coming soon (I am using it; but I am cleaning up and adding unit tests before I submit for review)

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Fri, 2005-01-28 00:23

here is the second part - restore!

The way to use it is as follows

  1. you make a backup of metadata on machine A
  2. you copy the metadata to machine B (obviously, A and B may be the same machine, especially for testing)
  3. you copy the album tree that corresponds to this metadata from A to be as well
  4. you run restore to get the albums into your gallery on machine B

Once Bharat reviews the backup part (nudge, nudge), I plan to do the following:

    Merge backup and restore into one module (can somebody suggest a decent name? Backup/restore? Export/import? Metadata?
    Make a better confirmation screen (it will require a little change in core - so it make take some time)
    Write more and better unit tests
    Respond to your suggestions - if there are any

[/]

[/]

[/]

[/]

[/]

 
mindless
mindless's picture

Joined: 2004-01-04
Posts: 8601
Posted: Fri, 2005-01-28 00:33

import/export makes more sense to me than backup/restore (which implies full-system backup to me..).
also it would be cool if the export packaged everything up into one file, ie metadata and image files.

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Fri, 2005-01-28 00:52

mindless, I thought about adding image files as an option (then it does become backup, doesn't it?). Here are my excuses why I haven't done it yet.

For me it is low priority - I am usually exporting data on my hosted machine, where space is at premium. So, I'd rather export metadata and move the files around separately.

I tried to get the results zipped and downloaded to the browser. Well, 'zipped' part works fine; however downloading to the client doesn't work. I looked how your zip download works (I even tried to encourage you to make it an API :P ) but the bottom line it doesn't work yet

By the same token, if everything is packaged and zipped together - import should be able to read from the archive. I saw a new archive upload module couple days ago - so I think this excuse isn't goinna work for much longer :x

The bottom line - either I will find another excuse, or I will do that as the next task.

 
Einstein
Einstein's picture

Joined: 2003-10-13
Posts: 105
Posted: Fri, 2005-01-28 09:58
virshu wrote:
mindless, I thought about adding image files as an option (then it does become backup, doesn't it?). Here are my excuses why I haven't done it yet.

But in a sence where your whole gallery installation is gone ... you haven't the gallery default settings saved. It's not a backup. I suggest also "Import/Export Album".

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Sun, 2005-01-30 06:08

here is the combined module. It's called metadata; and the option at the navigation links is 'export album'..

To import into another album, click on "add items" and there is a new tab called "From Backup" (somehow 'from export' doesn't sound right). Anyway, you have to have a direcotry tree you can import the metadata along with the pictures themselves.

Give it a try and let me know what you think.

There are still couple things that I am procrastinating...
1. Sending the results to hte browser (there are several activities around immediate view; I hope I will leverage those)
2. Adding the pictures to the export - not just metadata. This will create a full self-contained package.
3. Import from an archived file (zip, etc.) - I need to make the new archive module work on my machine first

 
bharat
bharat's picture

Joined: 2002-05-21
Posts: 7994
Posted: Thu, 2005-02-03 03:02

Sorry to take so long to get around to reviewing this! It looks like a good start -- but there's still a lot more to be done (but I'm guessing you already knew that). Here's a first round of comments. After you clean this up, I'm sure I'll have more...

General

  • Is BackupDataParser the best name? You can use it to do more than just parse backup data.
  • "metadata" isn't the best name for the module since that implies that we're managing metadata here, but we're really not. How about "importexport" instead?
  • your unit tests look really good. there are a couple of places where I think you need a little more, but in general I am very happy to see this level of detail!
  • BackupConfirmControllerTest.class tests ConfirmController <---those names should match up, one way or the other.
  • You're creating a new data format that has no versioning and doesn't associate logical names with the data fields, so as soon as we change something and you want to rev to a new version you're going to have a little difficulty figuring out what version of the data you're importing. I'd suggest that you at least associate a version number with the data file just to make your life easier when the next rev rolls around.
  • I make a bunch of comments below about using GalleryCoreApi::addExistingItemToAlbum -- that's because I reviewed ItemAddFromBackup last and didn't see where you were GalleryCoreApi::addItemToAlbum. The key issue here is to make sure that you don't duplicate the code that is in those helper methods, because you'll never be able to keep it in sync. We should use those two methods as much as possible, instead.

BackupDataParserTest:

  • 70: extra space at the end of the line
  • testLoadAlbumFields -- you should assertEquals *all* the fields. Otherwise, you won't know if one got silently dropped.
  • testCreateNewAlbum() only verifies that the new album that was created is parented correctly. However, the createNewAlbum() function does *way* more. Also, you should make sure that your new album has a thumbnail, an order weight, and other stuff. createNewAlbum() isn't doing any of that (hint: see GalleryCoreApi::addExistingItemToAlbum). You should also verify the layout, theme, title, etc data got imported correctly.

ItemAddFromBackupTest:

  • in setUp() can you use $this->_createRandomUser() instead of creating your own user by hand? I see that you care about the length of the username in your embedded data. But using a hardcoded username makes the test brittle (if the test fails, the user might stick around, then further test runs fail too because of a conflict). Instead you could use _createRandomUser and then pass the username as an argument to the constructor for your platform then correct the template to use the right run length for the length of the username.
  • 65: > 100 chars
  • You can use $this->_markForCleanup() on your user and get rid of your tearDown
  • 103: (and other lines) why not just hardcode 'unittest1' into GalleryUtilities::putRequestVariable() on line 104? This pattern is repeated in all your tests

BackupConfirmControllerTest:

  • the tests don't really do all that much to assert that the data got dumped properly. We should try a little harder to make sure that the correct export really happened.

Confirm.inc:

  • Don't use abbreviations. ("cnt", "toc", "includesub")
  • 35, 36, 37: use correct phpdoc style for these comments
  • 68: capital 'F' in first.
  • 78: don't use 'albumdb.dat' -- that's the name of a file from G1 and it'll cause unnecessary confusion (people will feed G1 albumdb.dat files in and complain when it doesn't work!)
  • Avoid using member variables when they're not necessary. _cnt is only used inside handleRequest() so it doesn't need to be a member
  • 83: you're not checking the return code from _serializeAlbum
  • 91: don't use ERROR_OUT_OF_SPACE here -- it's not what that error is for, and we don't know for sure that this is what went wrong in this case. I'd return ERROR_PLATFORM_FAILURE instead
  • 97: missing $ret check
  • 99,100,106: getcwd, chdir and glob must be called via the GalleryPlatform. I'd try to avoid changing the working directory if you can. It's bound to lead to trouble and usually isn't absolutely necessary.
  • 105: missing $success check
  • _traverseAlbums
    • General comment: separating functionality like this into its own method is great, but you break the methods separation when you have it store data in member variables. If, instead, you have it return the list of ids then it becomes its own separate method and can be moved into a helper. Either way, this is a perfect candidate for a unit test.
    • Doing this recursively does not scale well. Instead, we should use the GalleryItemAttributesMap::parentSequence to let you get *all* of the sub-albums in a single query. See ImageBlockHelper::setDisabledFlag() for an example of this
  • _serializeAlbums
    • See my comment above about properly separating helper methods. This should be a separate method with its own well-defined API governing inputs and outputs and its own set of unit tests.
    • 164: should be indented to 8 columns
    • I'd take this code in a different direction. I'd create a new interface called "ImportExportPlugin" and then let different modules implement it. For export, your framework would then let every implementation of the interface examine the data object and create its own chunk of output. And then do the reverse at input time. The core module can register its own implementation which would do all the basic stuff that it knows about. This will make it easy for other modules to tie in as appropriate. It's actually not that much more code and it makes what you have way more modular.

BackupDataParser:

  • 46-47: don't use albumdb.dat as the filename
  • 71-78: should be indented to 8 cols
  • 94: > 100 chars
  • 135: use single quotes
  • 141,149: use true/false instead of TRUE/FALSE
  • 188, 189: capitalization is off, should be "setCrea..." and "setVie...". Same for the getters inside the method.
  • 195: as I hinted about above, we should be using GalleryCoreApi::addExistingItemToAlbum() on the new album which does the necessary housework when you add an item to an album

Results.tpl:

  • 12,20: should be indented to 8 columns
  • 34,40: should be indented to 6 columns

ItemAddFromBackup.tpl:

  • 19-24: should be indented to 6 columns
  • 35: grammar fix: should be "...and contains a file called"
  • 35: as mentioned above, don't use albumdb.dat as the filename
  • 96: should be indented to 12columns
  • 113-116: should be indented to 8 columns

Confirm.tpl:

  • 40: should be indented to 6 columns.
  • 48: should be 4 cols
  • 40,49: I usually put the /> on the next line for clarity when I have logic inside the <input> tag.
  • 53: Explain a little better what isn't working yet.

ItemAddFromBackup.inc:

  • in general, you're doing too much work here. The form[action][addFromBackup] code should be moved into a helper class as a method with a clearly defined API that has nothing to do with controller logic. Then the controller should call this method as appropriate and communicate back success/failure.
  • 33, 34: use GalleryCoreApi::relativeRequireOnce
  • 64: hilites -> highlights
  • 66: ID -> Id
  • 76: > 100 chars
  • 93: missing space after the comma
  • 98, 101: (and elsewhere) targetAlbumID -> TargetAlbumId
  • 144: Don't set the order weight by hand, instead use GalleryCoreApi::addExistingItemToAlbum and let it do the housekeeping, then override anything afterwards that you want to be different. That way if we introduce some new bit of framework your code won't break.
  • 190: > 100 chars
  • 424: this should be localized

module.inc:

  • 59: The API has changed yet again. factory implementations should now be registered in performFactoryRegistrations().

[/]

[/]

[/]

[/]

[/]

[/]

[/]

[/]

[/]

[/]

[/]

[/]

[/]

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Mon, 2005-02-07 04:54

Bharat,

As usually, thanks for such detailed review!

I think I made all simple corrections... I have some questions/concerns about medium changes. And I need help on the complex enhancement.

  • I am using GalleryItem:save for my imported album, and it works really nice with one exception. DatabaseStorage:saveEntity only saves the members that have MEMBER_MODIFIED flag on. However, I need to save all members, since I just imported them from external storage (I can also see the same problem in the situation when you would try to copy an album from one parent to another - functionality, I don't think exists today). So, I would like to add a flag, $force in DatabaseStorage:saveEntity, and trickle it down to GalleryStorage:saveEntity, GalleryEntity:save, and GalleryItem:save.
  • If I read the framework code correctly, GalleryItem:create (which is not a static function, but is rather called as $album->create($parentId, $desiredPath) essentially works as $album->addMeToParent($parentId). At least, I use it in this way, and it works quite well. As such, it makes more sense to retrieve the pathComponent from $album itself, and possible update it if it collapses with existing subalbum/subdirectory.
    True, it is usually called for the album that was just created and doesn't have pathComponent, but this calling pattern should be transparent to the API itself. At least let's make this argument optional, and if it isn't provided, get the value from $this->getPathComponent().
  • another issue that I came across is setting relative weight in the parent. createAlbum() sets the weight of newly created album in the parent (which makes it appear at the end); however, create() doesn't. So, imported albums end up in the beginning of the parent. I don't have preferences one way or another; however for consistency it should be better to have the same weight-setting action
  • GalleryPlatform is missing glob() function. Shall I just add it?
  • As we discussed earlier, I want to have ItamAdd:handleRequest to not hardcode redirect destination (core:ItemAddConfirmation) but to allow plugin to supply it. Something like:
    /* if plugin provides its confirmation template - oblige */
        $subView = empty($status['subViewRedirect']) ? 
            'core:ItemAddConfirmation' : $status['subViewRedirect'];
        $results['redirect'] = array('view' => 'core:ItemAdmin',
    					 'subView' => $subView,
    					 'itemId' => $item->getId());
    

The complex change that I need some help with is this

Quote:
I'd take this code in a different direction. I'd create a new interface called "ImportExportPlugin" and then let different modules implement it. For export, your framework would then let every implementation of the interface examine the data object and create its own chunk of output. And then do the reverse at input time. The core module can register its own implementation which would do all the basic stuff that it knows about. This will make it easy for other modules to tie in as appropriate.

It may be not that much more code but I need some guidance on that.

As you can imagine, I implemented and tried the suggestions that I listed above (some even have unit tests). Assuming that you agree, what would be the best way to send them to you for review? I saw some people send diffs; but when you have just two words ($force in the list of parameters and in the parent call) I am afraid it will make your life too complex. Is there a better way?[/]

 
mindless
mindless's picture

Joined: 2004-01-04
Posts: 8601
Posted: Mon, 2005-02-07 06:22

if you add glob() you need to check if function_exists and provide something that will work if it doesn't (php 4.1.x, 4.2.x)

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Mon, 2005-02-07 06:35

hmmm, then I have the same problem with just calling glob(), as I am doing now. Let me replace it with something else (I will read manual one of these days! 8) ). OK, nevermind that bullet.

mindless, any comments on other topics?

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Mon, 2005-02-07 19:48

It turned out that glob wasn't that neccessary to begin with... recursiveRmDir worked just fine!

Another topic (that I didn't mention before) - concern about chdir. The alternative is to run 'zip archive.zip /full/path/to/the/files' which will add the directory structure to the archive. I could use a switch - 'zip -D archive.zip /full/path/to/the/files', but I don't think it's universal. So, it's balancing, what's worse, I guess!

Needless to say, I borrowed that snippet from zip download module; whatever the arguments were there - applies here as well. :wink:

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Mon, 2005-03-07 03:37

here is the module for the beta. Note, that it is going to have significant changes soon (again)

 
Thomm

Joined: 2003-11-02
Posts: 52
Posted: Mon, 2005-03-07 15:15

Thanks for the new version!!

 
moguai_

Joined: 2005-06-03
Posts: 2
Posted: Sun, 2005-06-05 11:31

is this version only for alpha gallery installations..?

i´m running beta3:

Quote:
Export and import album metadata
Incompatible module!
Core API Required: 4.0 (available: 5.8)
Module API Required: 0.10 (available: 0.11)

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Sun, 2005-06-05 11:44

moguai_, see:
http://gallery.menalto.com/index.php?name=PNphpBB2&file=viewtopic&p=144400#144400
in short: yes it only works for very old G2 versions. An update is coming soon.

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Sun, 2005-11-06 20:27

Hi all

Any news on this?

I just ran into a major crash disk last week... It took me quite some time to save what could be saved from my hard drive (fortunately, the MySQL databases were OK), to understand the SELinux security model and recreate all pictures I lost (from the originals stored on my main machine)...

As a result, I now plan to make backups of my Gallery on a regular base (something I did not do before)... So now you understand why this module is of interest to me.

By the way, it would be really nice if it could be called from the shell, so that cron can make backups and put them in a safe place.
If this module is not going to be GA soon, I'll just backup g2data/config.php/.htaccess with (s)tar and then the MySQL database with the appropriate tool.

 
virshu
virshu's picture

Joined: 2003-09-13
Posts: 314
Posted: Tue, 2005-11-08 00:37

I thought it is still working (except zip)... I am using it on a regular basis; not as much as backup, but more of a metadata export/import tool. Let me know if there are any issues; and I'll fix them. I'll also make sure that the versions are updated once I have access to development environment!

That said, for full backup nothing beats mysqldump or similar :) !

 
cosmicelf

Joined: 2005-08-21
Posts: 153
Posted: Thu, 2007-12-27 21:41

I realize this is a really old thread.

What is the best way for me to bulk export images from gallery 2 into a folder that I can zip up and give them to someone who wants to use them as a slideshow on a projector.

I would love to export the whole tree of events gallery including the "gallery tree" of all sub galleries as folders.

is this possible?

 
cosmicelf

Joined: 2005-08-21
Posts: 153
Posted: Fri, 2007-12-28 00:51

oops.. i had not really looked at the gallerydata file tree recently and did not realize that it stores them all with the same name you upload them in.

great. you can disregard my question.

peace