Model based validation is here (along with a REST API implementation)

Update: I have pushed this code to the main git repo as of 8 PM PST on 27-Jan-2010.

Hi, everybody. I've been heads down for the past few weeks working on a substantial change to the way that our model code works. In the next few days I'm going to push it to everybody so I thought I'd give you guys a chance to see what's coming and discuss. I've also updated all the code in contrib to match the new patterns so you shouldn't have to do anything here except read and understand

Important note: this change does not affect application security. Models don't enforce application security (like view and edit permissions) because they are independent of the active user. Both before and after this change, your controllers will still have to make the necessary access::can() and access::required() calls to make sure that the user is not taking an action for which they are not authorized.

In the old code, to add/edit/delete a model you'd create a form, and that form would have validation rules. You'd validate the form, and when all the fields are ok you'd update the model and save it. Here's an example of that from modules/gallery/controllers/albums.php:

THIS IS THE OLD WAY

    63      $form = photo::get_edit_form($photo);
    64      $valid = $form->validate();
    65  
    66      if ($valid) {
    67        $new_ext = pathinfo($form->edit_item->filename->value, PATHINFO_EXTENSION);
    68        $old_ext = pathinfo($photo->name, PATHINFO_EXTENSION);
    69        if (strcasecmp($new_ext, $old_ext)) {
    70          $form->edit_item->filename->add_error("illegal_extension", 1);
    71          $valid = false;
    72        }
    73      }
    74  
    ..        /* 25 lines of code to check for unique filenames left out here */
   100
   101      if ($valid) {
   102        $photo->title = $form->edit_item->title->value;
   103        $photo->description = $form->edit_item->description->value;
   104        $photo->slug = $form->edit_item->slug->value;
   105        $photo->rename($form->edit_item->filename->value);
   106        $photo->save();
   107        module::event("item_edit_form_completed", $photo, $form);

I left out some stuff to make it smaller, but you can see the whole thing at http://github.com/gallery/gallery3/blob/20bd09ff004816ae152a2f890a24dc5e85741fac/modules/gallery/controllers/photos.php#L63

Notice that we validate the form in line 64, then do more checking starting on line 66 if the form was valid, then do the actual saving starting on line 101. Whatever we set into the model on lines 102-105 get saved on line 106. The form draws rules from the model using an extension we made to Forge, but it also provides its own extra rules. And clearly we have some hand-written rules in the controller itself.

This worked just fine for us for a long time. But it has problems:
1) There's a lot of duplication between similar controllers. Like the code to edit albums and movies has to do some of the same checking so we had some cut/paste code. We don't like cut/pasted code. Bad!

2) Any other controller or helper that wants to create/update/delete a model has to have the same exact checking, or you run the risk of sticking bad data into the database. So for example if we made the title field mandatory we'd have to go to every controller and update its form. This leaves lots of wiggle room for mistakes. Bad!

We're now moving to a new approach called Model based validation. Right now this code lives in the bharat_dev branch on github, but it's got passing unit tests and is undergoing a code review so expect it to go live in the next day or two. Basically the way this works is that the model itself is responsible for making sure that all of its data is legal. If the data isn't legal, the model won't let you save it. Here's what the new code looks like (also from photos.php)

THIS IS THE NEW WAY

    63      $form = photo::get_edit_form($photo);
    64      try {
    65        $valid = $form->validate();
    66        $photo->title = $form->edit_item->title->value;
    67        $photo->description = $form->edit_item->description->value;
    68        $photo->slug = $form->edit_item->slug->value;
    69        $photo->name = $form->edit_item->inputs["name"]->value;
    70        $photo->validate();
    71      } catch (ORM_Validation_Exception $e) {
    72        // Translate ORM validation errors into form error messages                                                                 
    73        foreach ($e->validation->errors() as $key => $error) {
    74          $form->edit_item->inputs[$key]->add_error($error, 1);
    75        }
    76        $valid = false;
    77      }
    78  
    79      if ($valid) {
    80        $photo->save();
    81        module::event("item_edit_form_completed", $photo, $form);

The first thing you should notice is that it's a lot shorter. That's because we moved all the extra checking code into the model. I'll provide some code snippets for that part lower down. But the key things to note are:

1) On line 65 we check to see if the form's valid. We do this because 3rd party modules may add their own extra form fields and their own rules. It's still ok to have form rules!

2) On lines 66-69 we move values from the form over to the model. Even if they're invalid! That's because we know that if there are any validation issues, they'll be caught by the model.

3) On line 70 we call validate a second time. This makes sure that the new values in the model are actually OK. If there's a validation error here, then it throws an exception which we catch and then on line 74 we move those validation errors back into the form.

4) On line 79 if we passed all validation errors, we call save (which incidentally is going to make sure that the model is still valid -- you can't cheat!).

If validation fails we just dump out the form like we did before (I don't show that part here because it's the same as it was before.

So magically we've removed a lot of code, right? Well not exactly. All that checking code has been moved into the model. So in the Item_Model you'll see code like this:

   717    public function validate($array=null) {
   718      if (!$array) {
   719        $this->rules = array(
   720          "album_cover_item_id" => array("callbacks" => array(array($this, "valid_album_cover"))),
   721          "description"         => array("rules"     => array("length[0,65535]")),
   722          "mime_type"           => array("callbacks" => array(array($this, "valid_field"))),
   723          "name"                => array("rules"     => array("length[0,255]", "required"),
   724                                         "callbacks" => array(array($this, "valid_name"))),

http://github.com/gallery/gallery3/blob/0efbfcbe61e26cb53cb0719ced3c5c436f128386/modules/gallery/models/item.php#L718

There's some boilerplate in there, but the important thing to note is that, for example, the name field calls the Item_Model::valid_name() callback to make sure that the field is ok. Remember that code in the controller that made sure that you can't change the file extension? That now lives in Item_Model::valid_name:

   772    public function valid_name(Validation $v, $field) {
   ...   
   781      if ($this->is_movie() || $this->is_photo()) {
   782        if ($this->loaded()) {
   783          // Existing items can't change their extension                                                                            
   784          $original = ORM::factory("item")->where("id", "=", $this->id)->find();
   785          $new_ext = pathinfo($this->name, PATHINFO_EXTENSION);
   786          $old_ext = pathinfo($original->name, PATHINFO_EXTENSION);
   787          if (strcasecmp($new_ext, $old_ext)) {
   788            $v->add_error("name", "illegal_data_file_extension");
   789            return;
   790          }

http://github.com/gallery/gallery3/blob/bharat_dev/modules/gallery/models/item.php#L772

But the advantage here is that we've moved 2 copies of this code from our movie and photo controllers into 1 place in the model. We've done this for every field of significance. So if you want to modify a model field you can go for it, and if the value is illegal the model will reject it when you try to save. I've gone through the Item_Model and tried to lock it down so that you can't make a mistake.

We've also gotten rid of the functions like Item_Model::move_to() that move the item to a new location. They break the pattern. Instead, if you want to move a photo to a new album, you'd write:

   $photo = ORM::factory("item", 35);  // where 35 is the photo id
   $photo->parent_id = $album->id;     // id 1 is the id of the root album
   $photo->save();

That's it. The photo is moved safely, or it throws an ORM_Validation_Exception if you got something wrong (like you used an invalid id). Note that it does not do permission checking. Permissions are a layer on top of model consistency.

The really great thing about this is that now we can add a whole new REST based API that allows us to do really cool things without having to duplicate all of our integrity checks. For example, here's a snippet of code that shows how we modify an item in the REST code:

    85    static function put($request) {
    86      $item = rest::resolve($request->url);
    87      access::required("edit", $item);
    88  
    89      $params = $request->params;
    90  
    91      // Only change fields from a whitelist.                                                                                       
    92      foreach (array("album_cover_item_id", "captured", "description",
    93                     "height", "mime_type", "name", "parent_id", "rand_key", "resize_dirty",
    94                     "resize_height", "resize_width", "slug", "sort_column", "sort_order",
    95                     "thumb_dirty", "thumb_height", "thumb_width", "title", "view_count",
    96                     "weight", "width") as $key) {
    97        if (property_exists($request->params, $key)) {
    98          $item->$key = $request->params->$key;
    99        }
   100      }
   101      $item->save();
   102    }

http://github.com/gallery/gallery3/blob/0efbfcbe61e26cb53cb0719ced3c5c436f128386/modules/gallery/helpers/item_rest.php#L85

The first thing you should notice is that there's no validation code to be seen. That's because it's all in the model! If there's a validation error, the REST framework code just sends that back to the client. And also, we don't let you change every field because some fields are too dangerous to change (like MPTT pointers). But we let you change a lot of fields -- more than the UI lets you do. If you change the parent_id, then you've just moved the photo or album to a new location. So it's super flexible and powerful, requires very little code and gets 'er done. I like it a lot.

Here's an example in the REST client of modifying an album's title field:

    33  $album
    34    ->set("title", "This is the new title")
    35    ->save();

http://github.com/gallery/gallery3-contrib/blob/6a3851bae272ae3d1f406095c3f97a4bdbd089eb/client/example.php#L32

It's really, really easy.

Hope you like it and I look forward to hearing your thoughts.

---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git

I've been working with the boiler plate code. So far I can only get the login to work and tags.

Question. When I tried to get a list of albums using www.domain.com/gallery3/rest/albums I can not get anything to return. The API spread sheet shows that "should" be valid. Has it changed? Do you have to supply the token when doing anything other than getting "tags"?

I so far was only able to perform some of the TAG operations.

I was also able to send credentials and get a token back.

when trying to get /albums or /users or /ect... I was not able to get any results. Wasn't sure how much of it was operational or not. That's why I was thinking maybe I was just off because of the API out of date.

Have an interesting windows desktop application in the works to work with Gallery3. Thought the REST would be an awesome way to link them together.

Dear Mr Bharat Mediratta,

In an ongoing project, we want to use the REST API on Gallery 3. After going thourgh "gallery3-contrib\client\example.php" carefully, we got stuck and need further help from you for following issues:

1) how to get all tags,
2) how to get all albums under an album (not neccssary the root),
3) how to get all images in an album,
4) it seems the search functionality in "example.php" returns all albums and images which contains the search term. Can we narrow the search down by set('type', 'photo') , which will not search again album name.

We appreciate if you can provide some working examples.

Have a nice day!

William

Dear Mr Bharat Mediratta,

We would like to change both "the thumbnail size" and "the resize limit". However the values we haved changed through "Admin"->"Setting"->"Advance" DONOT have any affects at all, although they were successfully written back to table "vars". On the other hand, if we directly change the values in table "graphics_rules", it is working.

Our question here is what is the relationship between table "vars" and "graphics_rules" in terms of "the thumbnail size" and "the resize limit". Can we go though "Admin"->"Setting"->"Advance" and make it working?

Thanks!

@RESTful: you can change those sizes in the theme settings, and then it will update the graphics rules. Look in: Admin > Appearance > Theme Options

@heiv: sorry, I don't understand what you're asking.
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git

@heiv: I see, you're asking for some form of versioning for our REST API. I agree -- that's useful. Since we haven't released the 3.0 final yet, we haven't been worried about trying to do strict versioning because we're still trying to make major changes, but after the final version ships we will definitely provide you with a version number for the API.
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git

I created this ticket so that we don't forget: https://sourceforge.net/apps/trac/gallery/ticket/1148
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git

ORM version and error!!!

We downloaded Gallery 3.0 Release Candidate 1 in this March. The version of Kohana is 2.4, what is the ORM version?
We were hit by an error : "Not unique table/alias: 'items'" while running through following codes:
$model = ORM::factory('item')->where(...);
$total = $model->reset(false)->count_all();
followed by
$model->find_all();

Without using "reset(false)", no error, but apparently all the conditions set up in where clauses will be cleaned up

Any idea?

What we are trying to do is some sorts of pagenation for all photoes, so we need
"total number of items(photos)" before figure out how many pages and what are the items in current page.

Following is the code piece:

line 1: $model = ORM::factory("item")->where("items.type", "=", "photo");
line 2: $total = $model->count_all();
line 3: $photos = $model->find_all();

What happended is: after line 2, $photos contains everything, i.e. albums and photos, in othe words, the where clause in line 1 was ignored. After checking some documentation, we found one possible solution where
line 2: $total = $model->reset(false)->count_all();
However, we were hit by an error : "Not unique table/alias: 'items'"

bharat, thank you for so great post, I have such question: is it possible to insert photo using php form?

That's a bug, and I can't reproduce it. :-/ Please file a ticket!
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git