Developing Log Module

BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Sat, 2006-07-08 01:19

I'm developing a log module and in the upgrade function I need to add a onloadhandler to all Albuns and all Photos so I do something like this:

function upgrade( $currentVersion ) {
 global $gallery;
 if( ! isset( $currentVersion ) ) {
  $query = '
   SELECT [GalleryEntity::id]
   FROM [GalleryEntity]
   WHERE [GalleryEntity::entityType] = \'GalleryAlbumItem\' OR
         [GalleryEntity::entityType] = \'GalleryPhotoItem\'
  ';
  list( $ret, $searchResults ) = $gallery->search( $query );
  if( $ret ) return $ret->wrap( __FILE__, __LINE__ );
  while( $result = $searchResults->nextResult() ) $itemIds[] = $result[0];
  while( ! empty( $itemIds ) ) {
   list( $ret, $items ) = GalleryCoreApi::loadEntitiesById( array_splice( $itemIds, 0, 100 ) );
   if( $ret ) return $ret->wrap( __FILE__, __LINE__ );
   $gallery->guaranteeTimeLimit( 60 );
   foreach( $items as $item ) {
    if( $item->hasOnLoadHandler( 'LogAccess') ) continue;
    list( $ret, $lockId ) = GalleryCoreApi::acquireWriteLock( $item->getId() );
    if( $ret ) return $ret->wrap( __FILE__, __LINE__ );
    list( $ret, $item ) = $item->refresh();
    if( $ret ) return $ret->wrap( __FILE__, __LINE__ );
    $item->addOnLoadHandler( 'LogAccess' );
    if( $ret ) return $ret->wrap( __FILE__, __LINE__ );
    $ret = $item->save();
    if( $ret ) {
     GalleryCoreApi::releaseLocks( $lockId );
     if( $ret ) return $ret->wrap( __FILE__, __LINE__ );
    }
    $ret = GalleryCoreApi::releaseLocks( $lockId );
    if( $ret ) return $ret->wrap( __FILE__, __LINE__ );
   }
  }
 }
 return null;
}

Now I have two problems:
- First this process takes to long (I have about 5000 photos) and there is no progress bar indicator in the admin activate module.
- Second, for this to work, I have to change to use databases lock, because using files locks, I receive an error: "failed to open stream: Too many open files" as it tries to open more than 1024 file descriptors.

For thinks like what I want to do, wasn't much easier if exists the GalleryEntity:load event? Or I'm not seeing the think in the best way?

 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Sat, 2006-07-08 02:14

I found the solution for my second problem: I have to make $storage->checkPoint() from time to time, to commit the changes.

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Sat, 2006-07-08 07:42

You have a very good understanding of our codebase. Yes, when reading your code I too thought that a GalleryEntity::load event would probably make more sense than our current onLoad handlers.
Anyway, so you have the problem that we don't have a progress bar when activating modules in the site admin page.

even more so, we recently switched to using AJAX for the "activate module" UI. so the general assumption is that this takes a very short time.

your upgrade / activate method takes a long time though. so maybe we should introduce an optional progress bar for the upgrade method of plugins.

as to a short term solution:
- what kind of logging information do you aggregate there?
- maybe add a maintenance task that adds the logging handler to existing entities instead of doing this during module activation time.

 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Sat, 2006-07-08 17:18

Thanks for your reply.
First, yesterday I figure it out and I implemented the upgrade part of the module in a maintenance task so now I have the progress bar indicator.
The problem also exists in the uninstall part to remove all LogAccess onloadhandlers. Making another maintenance task is not the solution, as user expect to click uninstall and all work be transparently done.

But I made another test, I patched the GalleryEntity.class and now I have the GalleryEntity:load event and I can handle it like any other event. This resolve all my install/uninstall problems and I think that is much more elegant. Why not add this event to the main core files in Gallery 2.2? Do you think that the performance would be much prejudicated?

Another thing, I implement all my stuff in my personal module, that have things like: download hi-resolutions from a backup shared server, etc etc, and now the log addon too.
Should I make a standalone Log Module? Anyone anytime ask for it?

Last but not the least, I'm writing the log information in files in the [g2data]/logs/%year%month.txt and I only save very basic information, right now something like this:
08/07/2006 @ 04:10:16 - admin:10.0.0.90 - 2006/ (album:1)
08/07/2006 @ 04:10:30 - admin:10.0.0.90 - 2006/paris/ (album:1)
08/07/2006 @ 04:10:44 - admin:10.0.0.90 - 2006/paris/ (album:5)
08/07/2006 @ 04:11:07 - admin:10.0.0.90 - 2006/paris/dat0071p.jpg (photo)
...
I'm not saving log information in the database because performance problems.
Any interest in a module like this? Or sugestions...

PS: (album:1) means this item is an album and the user is seeing the first page.

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Sat, 2006-07-08 21:11

events are currently too expensive to add them for each entity load.
once the events framework is more lightweight, we could do something like that.

 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Mon, 2006-07-10 04:13

valiant, I need your help!
I'm trying to add an onLoadHandler as soon as an album is created, so I'm thougth in doing something like this:

function handleEvent( $event ) {
 if( $event->getEventName() == 'GalleryEntity::save' ) {
  $item = $event->getEntity();
  if( GalleryUtilities::isA( $item, 'GalleryAlbumItem' ) &&
      $item->testPersistentFlag( STORAGE_FLAG_NEWLY_CREATED ) ) {
   list ($ret, $lockId) = GalleryCoreApi::acquireWriteLock( $item->getId() );
   if( $ret ) return array( $ret->wrap(__FILE__, __LINE__), null );
   $ret = $item->addOnLoadHandler( 'LogAccess' );
   if( $ret ) return array( $ret->wrap(__FILE__, __LINE__), null );
   $ret = $item->save();
   if( $ret ) return array( $ret->wrap(__FILE__, __LINE__), null );		
   $ret = GalleryCoreApi::releaseLocks( $lockId );
   if( $ret ) return array( $ret->wrap(__FILE__, __LINE__), null );
  }
 }
}

But this is not working, it always give-me ERROR_STORAGE_FAILURE error!

Where should I implement my addhandler code for a newly created album ?

Thanks any help.

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Mon, 2006-07-10 06:27

what are the storage error details? you should get details, if logged in as admin.
for more error details, you can enable buffered debug mode.

FAQ: How to set/use Gallery in debug mode?

 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Tue, 2006-07-11 16:42

I always get the same error (only change the itemId) if I use the $item->save() function in the event handler function,
An example on a very simple code:

function handleEvent( $event ) {
 if( $event->getEventName() == 'GalleryEntity::save' ) {
  $item = $event->getEntity();
  if( GalleryUtilities::isA( $item, 'GalleryAlbumItem' ) &&
      $item->testPersistentFlag( STORAGE_FLAG_NEWLY_CREATED ) ) {

   list ($ret, $lockId) = GalleryCoreApi::acquireWriteLock( $item->getId() );
   if( $ret ) return array( $ret->wrap(__FILE__, __LINE__), null );
									
   $ret = $item->save();
   if( $ret ) return array( $ret->wrap(__FILE__, __LINE__), null );
					
   $ret = GalleryCoreApi::releaseLocks($lockId);
   if( $ret ) return array( $ret->wrap(__FILE__, __LINE__), null );

  }
 }
}

A get this error:

1062: Duplicate entry '24' for key 1

adodb_mysqlt._execute(INSERT INTO g2_AccessSubscriberMap (g_itemId, g_accessListId) VALUES (24,9)) % line  836, file: adodb.inc.php
                           
adodb_mysqlt.execute(INSERT INTO g2_AccessSubscriberMap (g_itemId, g_accessListId) VALUES (?,?), Array[2]) % line  985, file: GalleryStorageExtras.class gallerystorageextras.addmapentry(GalleryAccessSubscriberMap,Array[2]) % line  501, file: GalleryStorage.class
                           
mysqlstorage.addmapentry(GalleryAccessSubscriberMap, Array[2]) % line 2923, file: GalleryCoreApi.class gallerycoreapi.addmapentry(GalleryAccessSubscriberMap,Array[2]) % line  294, file: GalleryEntity.class
 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Tue, 2006-07-11 16:54

I'm calling a save function of an item that is already inside a save function (because I'm inside a save event handler), can't I do that? If I can't do that, how can I test if it has and add if it is the case an item onLoadHandler?

 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Tue, 2006-07-11 17:04

I'm calling a save function of an item that is already inside a save function (because I'm inside a save event handler), can't I do that?
I understand that this will cause a loop save event, so how can I add a onLoadHandler on an album as a save event is triggered ?

 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Tue, 2006-07-11 18:50

I read a lot of the Gallery and Modules source code, and I don't find a way to change an AlbumItem (example: to add a onLoadHandler) as soon as the Album is created.

I found 2 ways that gallery give a module to process items as soon as they are added/edited: postEvents and the handleRequestAfterAdd/handleRequestAfterEdit methods.

To add a onLoadHandler to an album as soon the album is created: I can't use the handleEvent with GalleryEntity:save because when this event is posted within an album creation I can't use the item->save method inside an already save method (problem in the previous posts).
The handleRequestAfterAdd in the ItemAddOption don't apply to AlbumItems.

So, how should I do? Any help?

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Tue, 2006-07-11 19:18

what about not calling $entity->save() in your event handler? GalleryEntity::save() does the save already for you.

PS: you'll have to ensure that you get the entity from the event by-reference.

 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Tue, 2006-07-11 20:11

I already had tried that way with this code:

function handleEvent( $event ) {

 if( $event->getEventName() == 'GalleryEntity::save' ) {

  $item =& $event->getEntity();

  if( GalleryUtilities::isA( $item, 'GalleryAlbumItem' ) &&
      $item->testPersistentFlag( STORAGE_FLAG_NEWLY_CREATED ) ) {
					
   $ret = $item->addOnLoadHandler( 'LogAccess' );
   if( $ret ) return array( $ret->wrap(__FILE__, __LINE__), null );
  }			

 }

}

But the handler is not added to the entity...

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Wed, 2006-07-12 08:26

ic, ::postEvent($event) doesn't use by-reference, neither does entity->setEntity($entity).
i think we should change that.
can you do the necessary changes in GalleryCoreApi::postEvent and in GalleryEvent::setEntity and ::getEntity?
make it always use by-reference.

 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Wed, 2006-07-12 13:09

If in the GalleryEvent::setEntity I change from:

function setEntity($entity)

to:

function setEntity(&$entity)

The Event constructor will fail as you cannot pass null by reference...

function GalleryEvent() {
 $this->setEventName(null);
 $this->setEntity(null);
 $this->setData(null);
}
 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Thu, 2006-07-13 07:28

please remove the setEntity(null) call from the constructor.

 
BeRnOiCo

Joined: 2006-05-20
Posts: 17
Posted: Thu, 2006-07-13 13:22

In GalleryCoreApi I changed

function postEvent($event)

to:

function postEvent(&$event)

In GalleryEventHelper_simple.class I changed

function postEvent($event)

to:

function postEvent(&$event)

In GalleryEvent.class I changed

function setEntity($entity) and
function getEntity()

to:

function setEntity(&$entity) and
function &getEntity()

and removed the setEntity(null) in the GalleryEvent constructer

But continues to not work... I dont know what to do any more...