synchronization cache and database

kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sat, 2010-08-07 11:11


Gallery version (not just "2"):2.3.1
PHP version (e.g. 5.1.6):
PHPInfo Link (see FAQ):
Webserver (e.g. Apache 1.3.33):Apache
Database (e.g. MySql 5.0.32):Mysql
Activated toolkits (e.g. NetPbm, GD):
Operating system (e.g. Linux):Linux
Browser (e.g. Firefox 2.0): all the browser

Hi, I'm italian and I don't know if I can explain in the right way our problem. We are the admins of twilightersitalia.com and we have installed gallery2.
but like one month ago we have deleted a folder of images directly from the ftp and the gallery didn't work from these day.
Our host, Aruba, said that cache of the gallery is not synchronized with the data and the images that are on the database MySQL, but they don't how to resolve it, so they tell us to go and ask to you...

could you help us??!!

thank you

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Sat, 2010-08-07 14:07

You should never delete or add images from or to, or otherwise directly interact with, your g2data folder.

Now that you have done this, your options are:

1. The easy way is to ask your host to restore that folder from their backup archives and then delete the album from within G2.
2. A more difficult way is to find out which images were in that album, recreate them and then delete the album from within G2. The creation can be a small jpg file of anything which you then duplicate and rename to match the original file names. You can use something like www.waybackmachine.org to go back to see what the album contained site was like before the deletion.
3. The hard way is to get a dump of all current Item IDs and then go through your database and manually delete any record that isn't in that list.

To get a dump of the items ids, take the code below and put it into your theme.tpl near the bottom just before the {g->debug}. It should list all the item ids of items within each album. Type them into excel, sort the list and delete duplicates. (IMPORTANT: Add item id '0' to your list ... i.e. do not delete item id '0')

{if $theme.pageType == 'album'}
	{if count($theme.children)}
		{foreach from=$theme.children item=child}
			<div>{$child.id}</div>
		{/foreach}
	{/if}
{/if}

Remove the code once you are done ... also remove whatever malware it is that Google says your site is distributing (http://google.com/safebrowsing/diagnostic?tpl=safari&site=twilightersitalia.com&hl=en-us)

If you can't navigate the site at all, then options 1 or 2 are about all I can think of that you can do.

--
dakanji.com

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sat, 2010-08-07 13:53
 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Sat, 2010-08-07 14:06

Nice one.

This assumes the user knows the Item ids of the missing items so will need to be combined with Option 3 above.

I suppose it could be expanded to recurse through the albums get the current ids, compare this with a list of those in the db and delete the extras.

OP will have to do this or hire someone to write this up for him. Hopefully he will post the resulting script here for possible pushing into the lib/support scripts.
--
dakanji.com

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sat, 2010-08-07 18:06

Ok, I've updated the script to search for lost or missing items automatically either by album, or recursively from root.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Sat, 2010-08-07 22:35

Cool.

Taking a quick look at it though, I think you may need to amend it a bit so that the HTML runs first and the user can input the path to G2.

At the moment, it is referencing global gallery and the configs in config.php as well as embed.php but can't know where these are located.

Structure probably should be a form asking for the path to G2 and then run "init(path_to_g2);" so that you can "require once path_to_g2 . embed.php;"

Also not sure where user will input specific id.

Well done all the same.

--
dakanji.com

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sat, 2010-08-07 22:46

No. Embed and config are located in their root of their G2 directory, which is where I instruct them to put it.
I actaully tested this. :)

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Sun, 2010-08-08 05:45

Ha, I see ... Wasn't aware.
Great work

--
dakanji.com

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sun, 2010-08-08 12:40

thank you so much, really!! **

but Aruba have already restore the gallery database, but nothing have changed...

so I wanted to try your script suprsidr, but where I can find the itemID for the folder that we have deleted from ftp?

thanks again!!

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sun, 2010-08-08 13:14

I also tried to put the code of Dayo in theme.tpl, but when I go to the gallery it shows only a blank page with Error...

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Sun, 2010-08-08 13:18

The theme.tpl thing will not work since g2 is down.

The idea with the script is that you put it into your G2 folder and navigate to it in your browser. It should then find the missing items for you and remove references to them from your db.

--
dakanji.com

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sun, 2010-08-08 13:32

and so the problem is: why the gallery is down?? we've already done the restore of the backup in the database...

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sun, 2010-08-08 13:50

I guess, maybe I'll add a form with an album list selector to let the user to choose where to start from.
Spiffy it up a bit too ;)

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Sun, 2010-08-08 17:50
kiucci wrote:
and so the problem is: why the gallery is down?? we've already done the restore of the backup in the database...

Restoring the database has no relevance to your issue which is that you manually deleted a folder. This is why your gallery is down.

If you go back and read through this thread, you will see that no one recommended restoring the database as a solution.

Run the script to fix the problem.
--
dakanji.com

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sun, 2010-08-08 17:58

sorry >.<

when I run the script, it shows:

GalleryEmbed::init failed, here is the error message: Error (ERROR_STORAGE_FAILURE)

* in modules/core/classes/GalleryStorage/GalleryStorageExtras.class at line 203 (GalleryCoreApi::error)
* in modules/core/classes/GalleryStorage.class at line 313 (GalleryStorageExtras::loadEntities)
* in modules/core/classes/helpers/GalleryEntityHelper_simple.class at line 71 (GalleryStorage::loadEntities)
* in modules/core/classes/GalleryCoreApi.class at line 2361 (GalleryEntityHelper_simple::loadEntitiesById)
* in init.inc at line 204 (GalleryCoreApi::loadEntitiesById)
* in modules/core/classes/GalleryEmbed.class at line 129
* in missingObjectFix.php at line 39 (GalleryEmbed::init)
* in missingObjectFix.php at line 114

??

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sun, 2010-08-08 18:11

I've never run it on a site that is down.
You'll need to fix the STORAGE_FAILURE first.
It is a database error.

I'm wondering if changing line 39 of missingObjectFix.php to:
$ret = GalleryEmbed::init(array('fullInit'=>false));
would help.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Sun, 2010-08-08 18:48
kiucci wrote:
sorry >.<

Sorry for coming across so harsh.

I now doubt if the script will work for you as it uses the G2 Embed (tries to create a G2 instance when the site is down). Additionally, your site has in all likelihood been hacked as your host, Aruba, has some bad outfits on their servers as per the Google warning and heaven knows what they have changed on your site.

Your best bet is to try the option involving the waybackmachine and hope your virtual neighbours have not vandalised the site and only added malware distribution to your setup.

You will of course need to get to the root of how the malware got put on your site (assuming you didn't put it there yourself) or you will get hacked again soon.

--
dakanji.com

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sun, 2010-08-08 20:58

oh my god, I didn't understand anything @.@

the script with the change doesn't work at all.

I'm searching in waybackmachine but there isn't the the last day with the folder that we deleted, it was the 21th June, but the waybackmachine stops at 30th April ç_ç

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sun, 2010-08-08 21:01

oh...sorry, but I want to thank you so much for the help **

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Sun, 2010-08-08 21:35

I think your problem goes beyond simply deleting that folder. As a last throw of the dice, recreate an empty version of the folder directly in location in g2data it was before and try running the script.

If this doesn't work then you probably have to get a new installation.

  • Create a new folder and copy all your old g2 files into it
  • Download a new copy of G2 to your webspace
  • Install the new G2 and create all your albums in G2 except the one you deleted since you don't want it anymore. You can look at the structure in g2data/albums.
  • use the add from server feature to repopulate them from your old g2data folder (again this has to be done in G2 ... remember you should not directly add or remove folders from g2data - the option above is an exception to fix a problem.

--
dakanji.com

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sun, 2010-08-08 22:15

I can try to recreate what folder? and where?

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Sun, 2010-08-08 22:25

The folder you deleted. Try to recreate it (empty) in the same location it was before deletion and then rerun the script.

If this doesn't work, and it is only a slim chance, try to rebuild the gallery with a new installation as described.

--
dakanji.com

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sun, 2010-08-08 22:42

but I have to recreate it in the database or in the ftp?

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Sun, 2010-08-08 22:46

my latest version v.0.6 skips the loading of any template info, so should connect to your gallery.
simply edit line 13 with your password and upload to your gallery root directory...

I just deleted an album directory on my test gallery and the script fixed it no problem.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Sun, 2010-08-08 23:04

wow, thank you so much, you're too kind, really!!!

now I have this situation:

http://i34.tinypic.com/2vlqwip.png

I think it's better from before, but before I have to solve the problems in the Error part...could you help me??

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Mon, 2010-08-09 00:00

Sorry, that's the same database error as before.
Do you have a database backup from this morning, before your host did your restore? Does your host have one from earlier today?

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 00:05

the oldest backup that I have is from 3rd August, but I don't think it can change so much, because all this story start on 21th June >.<

there's no way to solve the errors?

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Mon, 2010-08-09 00:08

I think you should restore the most recent backup.
Your gallery was accessible before your host restored for you correct?

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 00:17

no, it doesn't work from 23th June (just today I have time think about it), even if we and our host have restored the database...

maybe the only thing to do is to get a new installation, if we can solve the Error (ERROR_STORAGE_FAILURE)

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Mon, 2010-08-09 08:07

kiucci
Put back that last backup of your database (03 August) and do not change or restore the database again.
Run the script after and see.

Wayne
I believe there is probably something else beyond the album deletion going on with his gallery.
The way to absolutely solve this would probably be to use customised version of G2 functions with error trapping.
I.E., an attempt to init gallery which traps ids that generate storage failure errors without stopping. These ids can then be deleted from the db.

--
dakanji.com

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 13:46

nothing is changed with the 03August backup...

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Mon, 2010-08-09 13:54

run the upgrader
twilightersitalia.com/gallery/upgrade

specifically step 2 system checks

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 14:16

I made the upgrade and nothing is changed again...

I have 2 warning in step 2:

1. for the output_buffering enable in php.ini
2. for the changes in some files in the theme of the gallery

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Mon, 2010-08-09 14:26

Do you have access to server error logs?
If not, ask your host for clues in the logs pertaining to your gallery.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Mon, 2010-08-09 14:50

I only just looked closely at the error screenshot you posted and can see that the bit where your gallery is crashing is actually not related to the missing folder and has to do with user records.

In simple speak, somebody has been playing around in your database and it is now messed up ... the deleted folder is just a distraction.

This is why the script works on suprsidr's test gallery but not on yours.

In order of importance, your issues are:
1. Your database is messed up and gallery cannot connect to it.
2. You deleted a folder in your g2data via ftp

You need to fix Item 1 before going to Item 2. Item 2 is easy to fix ... the script does this.
Item 1 is another matter and this has to be fixed first. My only suggestion is that you restore the last db version before you started having issues. Once you have a working G2 (at least one that loads), you can run the script.
If you cannot get an undamaged database, you will need to reinstall G2.

--
dakanji.com

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 14:50

I can see just the most recent error logs and about the gallery there is only this typw of error:

[Mon Aug 09 16:44:02 2010] [error] [client 207.46.204.177] ModSecurity: Warning. Pattern match "msn(?:bot|ptc)" at REQUEST_HEADERS:User-Agent. [file "/etc/apache/modsecurity/modsecurity_crs_55_marketing.conf"] [line "17"] [id "910008"] [msg "MSN robot activity"] [severity "NOTICE"] [hostname "www.twilightersitalia.com"]

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Mon, 2010-08-09 14:56

@Dayo, so you're thinking user 5(Guest) or 6(Admin) has been deleted?
I wonder if we could simply insert those records.

But if there are not too many items in your gallery, re-install may be less trouble.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 15:01

not too many? XD just 24560 photos ç_ç

you think I can insert the record??

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 15:10

actually now that you tell me about the users, I don't have a user table in database O.o

there is just g2_pendinguser... I don't know it is normal...

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Mon, 2010-08-09 15:17

You can try:

DROP TABLE IF EXISTS `g2_User`;
CREATE TABLE IF NOT EXISTS `g2_User` (
  `g_id` int(11) NOT NULL,
  `g_userName` varchar(32) NOT NULL,
  `g_fullName` varchar(128) default NULL,
  `g_hashedPassword` varchar(128) default NULL,
  `g_email` varchar(255) default NULL,
  `g_language` varchar(128) default NULL,
  `g_locked` int(1) default '0',
  PRIMARY KEY  (`g_id`),
  UNIQUE KEY `g_userName` (`g_userName`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO `g2_User` (`g_id`, `g_userName`, `g_fullName`, `g_hashedPassword`, `g_email`, `g_language`, `g_locked`) VALUES (5, 'guest', 'Guest', 'U7T4b19b16ff9d1f6e1e22b686394e8d4c42', NULL, NULL, 0),
(6, 'Admin', 'Gallery Administrator', 'A3gs01c022fdf6d2e82d99866eea3ce1064d', 'youremail@youremail.com', NULL, 0);

Adjust the email address in the above insert. Then you'll have to use the password recovery if we get your gallery backup.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Mon, 2010-08-09 15:22

@kiucci
These are the tables I have in my installation. Does it match yours?

 g2_AccessMap
 g2_AccessSubscriberMap
 g2_AlbumItem
 g2_AnimationItem
 g2_CacheMap
 g2_ChildEntity
 g2_DataItem
 g2_Derivative
 g2_DerivativeImage
 g2_DerivativePrefsMap
 g2_DescendentCountsMap
 g2_Entity
 g2_EventLogMap
 g2_ExifPropertiesMap
 g2_ExternalIdMap
 g2_FactoryMap
 g2_FailedLoginsMap
 g2_FileSystemEntity
 g2_Group
 g2_ImageBlockCacheMap
 g2_ImageBlockDisabledMap
 g2_Item
 g2_ItemAttributesMap
 g2_Lock
 g2_MaintenanceMap
 g2_MimeTypeMap
 g2_MovieItem
 g2_PermissionSetMap
 g2_PhotoItem
 g2_PluginMap
 g2_PluginPackageMap
 g2_PluginParameterMap
 g2_RecoverPasswordMap
 g2_Schema
 g2_SequenceEventLog
 g2_SequenceId
 g2_SequenceLock
 g2_SessionMap
 g2_ThumbnailImage
 g2_TkOperatnMap
 g2_TkOperatnMimeTypeMap
 g2_TkOperatnParameterMap
 g2_TkPropertyMap
 g2_TkPropertyMimeTypeMap
 g2_UnknownItem
 g2_User
 g2_UserGroupMap

I don't have a g2_pendinguser but there is a g2_user. I think g2_pendinguser applies if you allow user registrations which I don't. One thing to do immediately is to change your ftp password, stop using ftp and use sftp instead if possible and make sure you don't have spyware on your machine especially if you are on Windows.

@suprsidr
Users can be inserted but this doesn't deal with the question of how they got deleted in the first place. Whoever deleted the record before can, and will, delete them again and next time probably do even more. FTP access can be dangerous and if the ftp password has been compromised, it is a simple matter for an intruder to place a php script with a "drop" sql command in.

--
dakanji.com

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Mon, 2010-08-09 15:24
suprsidr wrote:
You can try:

DROP TABLE IF EXISTS `g2_User`;
CREATE TABLE IF NOT EXISTS `g2_User` (
  `g_id` int(11) NOT NULL,
  `g_userName` varchar(32) NOT NULL,
  `g_fullName` varchar(128) default NULL,
  `g_hashedPassword` varchar(128) default NULL,
  `g_email` varchar(255) default NULL,
  `g_language` varchar(128) default NULL,
  `g_locked` int(1) default '0',
  PRIMARY KEY  (`g_id`),
  UNIQUE KEY `g_userName` (`g_userName`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

If he does this, he will lose all his user records
--
dakanji.com

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 15:28

my tables end at g2_SequenceEventLog -.-'

so HERE is the real problem >.<

ye, we have user registration and I'am the only admin of the staff who use Macintosh, the others have Windows.

I changed the ftp password yesterday for the malware attack of the past days

how can I use sftp?? I use Filezilla to work on files...

and now??!!

 
Dayo

Joined: 2005-11-04
Posts: 1642
Posted: Mon, 2010-08-09 15:50

You need to find a backup of your db from working g2 installation on your site and load the missing tables. If not available, You can try running suprsidr's sql and then adding blank versions of the others. Most likely, you will need to reinstall.

Filezilla has sftp (secure ftp). Your host may not permit you access this way though ... you need to ask.

BTW. If you or your staff have spyware that is broadcasting your ftp password to some hacker in Siberia, they already have your new one already.

Change it again today and tomorrow and the day after .... until each machine has been throughly scanned. After that, change it every week or at least frequently. Your guys probably log in at home sometimes and God only knows what they have on their home computers.

--
dakanji.com

 
suprsidr
suprsidr's picture

Joined: 2005-04-17
Posts: 8339
Posted: Mon, 2010-08-09 15:48

I can continue to feed you tables and inserts, but looks like you'll have to re-install your toolkits.

-s
FlashYourWeb and Your Gallery with The E2 XML Media Player for Gallery2

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 16:15

ok I found a working backup and restored it, and now the gallery and your script work (I just want to say that Aruba said he had restored the same backup that I restored today -.- ), but only for the admin; for the users it shows "Closed for working"...

I run the script for the album where was the folder that we have deleted, but nothing changed...

 
kiucci

Joined: 2010-08-07
Posts: 20
Posted: Mon, 2010-08-09 16:18

it works!!! *_____*

wooooow, thank you so much!!!!

I really don't know how to thank you, both ^^