Research and Experience gained

It's been a long time, my poor little blog (and it's followers(s)). A lot of my work is now going to be company confidential. But I will share what I have figured out . . . if it's already out on the web somewhere.

1) Most Server site software development frameworks now use URL rewriting.URL rewrite can 'scrape' variable/value pairs out of apparent directories after a site address. For example:

http://www.site.tld/variable-A/value-A/animal-type/dog/

The delimiters can be fairly custom within the allowed URL character set. For example:

http://www.site.tld/variable-A#value-A/animal-type#dog/
http://www.site.tld/variable-A#value-A/animal-type#dog/breeds#poodle;afghan;border-collie/

2) When using a modern software development framwork like Symfony, Ruby on Rails, .Net MVC, and the Java MVC products, one of the VERY FIRST THINGS THAT YOU WANT TO DO between doing all the business modeling, use cases, and other top level software design tasks, and actually coding is to map the resources and modules to URLs. Part of that is deciding what formats to supply upon request and where to signify the format on the URL.


3) If a site is going to use multiple formats, the most common way for that to to the server what format is desired is to use a file extension, but use it ubiquitously.

That means, as normal, if a URL looks like:

http://subdomain.domain.tld/directory/filename.ext

Just change the extension to what is desired, and if the site supports it, grand. Usual suspects are .rss|.xml|.json|.html. The ubiquitous part means NO MATTER WHAT IS THE LAST MAJOR VALUE AT THE END OF THE URL (except for variable/value pairs and query strings), PUT THE '.format' AT THE END OF IT. Examples will show it clearly:

http://subdomain.domain.tld/module-or-resource/action.FORMAT

http://subdomain.domain.tld/fake_directory_name_for_user_readability/module-or-resource/action.FORMAT

http://subdomain.domain.tld/module-or-resource/action.FORMAT/var1#value1/var2#value2/

http://subdomain.domain.tld/fake_directory_name_for_user_readability/module-or-resource/action.FORMAT?not-user-friendly-query-string-value=something&also-serach-engines-dont-catalog-this-full-url=but-we-want-that

http://www.site.tld/resource/action/id.format

http://www.site.tld/products/ship/4d912f22c182293a70e2e7ac3671228dff397a52.FORMAT/carrier#ups/rate#blue/address#123-Mocking-Bird-Lane-Uphigh-CO-80230-USA/currency#usd


The '.format' is between the end of the part of the URL that selects the resource or code and the part of the URL that supplies variables to the code processing the request. (PHP/Symfony, PHP/Drupal, Ruby/Rails, et al. all use this)

This convention/practice was decided upon in my projects after going through the research below. I hope it is as useful to you as it was to me:

=========================================

=================
I looked at the following sites:
Twitter
yahoo
amazon S3
facebook
myspace
linkedin
google.

(EOL = End of the Last listed of Collection, Module, Action ,Or Id)

The methods of specifying the return type of the document were one of three different kinds:

1/ Implied, because only one was available (Amazon S3:json, Linkedin:xml)
2/ Implied but another could be specified BY ADDING ‘.format’ TO THE END OF THE EOL before any query string or friendly URL components at the end of the whole URL (MySpace-xml:add.json)
3/ Implied but another could be specified by a query string paramter (google-atom:‘alt=json’, yahoo-xml:‘output=json’)
4/ Format was required BY ADDING ‘.format’ TO THE END OF THE EOL ((Twitter:add .json|.xml|.rss)

=========================================

API Versions numbers used in URL of the the API for:
Yahoo
MySpace

EXAMMPLES of the way '.format' is added to API URLs:
-------------------------------------------------------
http://microformats.org/wiki/rest/urls
HTML
 GET /people/1
     return the first record in HTML format
 GET /people/1.html
     return the first record in HTML format
XML
 GET /people/1.xml
     return the first record in XML format
JSON
 GET /people/1.json
     return the first record in JSON format

http://confluence.sakaiproject.org/display/SAKDEV/EntityBroker+RESTful+URL+support

Access to an entity:(MEMBER)

   * http://localhost:8080/direct/webapp-entity/id0
   * http://localhost:8080/direct/webapp-entity/id0.xml
   * http://localhost:8080/direct/webapp-entity/id0.json

Access to an entity space:(COLLECTION)

   * http://localhost:8080/direct/webapp-entity
   * http://localhost:8080/direct/webapp-entity.xml
   * http://localhost:8080/direct/webapp-entity.json

Describing entities: (SELF-DISCOVERY/ERD)
   o http://localhost:8080/direct/describe
   o http://localhost:8080/direct/eval-evaluation/describe
   o http://localhost:8080/direct/webapp-entity/describe
   o http://localhost:8080/direct/webapp-entity/describe.xml

MySpace
EXAMPLE REQUEST:
   * XML: http://api.myspace.com/v1/users/454304609/albums
   * JSON: http://api.myspace.com/v1/users/454304609/albums.json

Twitter
http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search
Search(GET)
 o Example: http://search.twitter.com/search.json?callback=foo&q=twitter
            http://search.twitter.com/search.atom?lang=en&q=devo
Statuses/User_timeline

http://twitter.com/statuses/user_timeline/12345.xml or http://twitter.com/statuses/user_timeline/bob.json.
http://twitter.com/statuses/user_timeline.xml?user_id=1401881
http://twitter.com/statuses/user_timeline.xml?screen_name=101010
http://twitter.com/statuses/user_timeline.rss?page=3
Status Updates (POST)
curl -u user:password -d "status=playing with cURL and the Twitter API" http://twitter.com/statuses/update.xml


 
Further References
http://www.xml.com/pub/a/2004/12/01/restful-web.html
http://ajaxpatterns.org/RESTful_Service (EXCELLENT)*  Describing entities:
   o http://localhost:8080/direct/describe
   o http://localhost:8080/direct/eval-evaluation/describe
   o http://localhost:8080/direct/webapp-entity/describe
   o http://localhost:8080/direct/webapp-entity/describe.xml

Symfony recognizes four main http methods

Well, it turns out that Symfony gets loaded in all the methods except HEAD, as expected.

I tried:
-----------
GET
POST
DELETE
PUT
HEAD
OPTIONS

Symfony reported the method correctly in the ones that really matter, the first 4 above. In Options, it displayed GET as the method. In HEAD, it was never invoked, which agrees with the HTTP specification, so that's a good thing.

Well, that DOESN'T change the fact that PHP won't parse the body for DELETE or PUT requests and so there is no pretty access to the body. I'd have to come up with my own MIME/POST processor. Not going there :-(

So, I will use the POST for all PUTS, and probably most multiple DELETES. As I had originally talked about in a previous blog.

However, in Symfony, I need to deal with the crsf variable in the body of the request, and NO direct GET variables in the URL. Mr. Ponticier (creator of Symfony) has added an optional,standard POST/body-of-the-request variable for 'method'. I may or may not use that. Investigating now.

PS., one other thing that I would have to deal with is Symfony's labeling of variables, they already use the brackets for their use of an array of values for columns of a table.

And I tested once, PHP doesn't accept multiple sets of brackets for arrays of arrays in the names of POST variables.
Investigating.

One step back

Well, I had the code for doing a REST-POST(single or arrayed) as I have previously described completely done. Then I learned what SVN and backups should be good for :-(

I found out a way to switch two files, do my normal neurotic 1 save/min, and end up with only one of two files that I had written. Since I had been working on it 4-6 hours a day, in addition to my full time job, I was exhausted and despondent after doing it.

So, I stepped back, relaxed, and decided to skip the temporary functional style design and go straight to Symfony. I am reading all I can on REST a la Symfony. I don't totally agree with what I've seen, and especially, from the founder of Symfony. However, he is much better at using his framework than I am ;-) So I will keep an open mind as I try to do it, 'my way'. (see previous posts on this blog)

I am just about to see start a micro project to test two things:
1/ How to use the crsf token that Symfony generates in a REST environment.
2/ If the sfWebRequest object contains the Method used to access the script, and see if I can actually at least use single JSON DELETE/POST.

Someday, I am going to pay someone to hack PHP and allow the $_POST variables to be filled during POST/DELETE/PUT/GET methods.

Implementing JSON REST with BLOBs in PHP - The practical way.

I have retrenched, somewhat, (if my memory of what I've been writing serves me correctly.)

I have decided to skip the use of the PUT method until PHP supports it better.

I would like to see full $_GET, $_POST, $_PUT, $_DELETE, $_COOKIE, $_OPTION, $_HEAD, $_REQUEST, etc request body support for the appropriate methods in PHP. From what I've read, this is within the full specs of the HTTP protocol. I'm pursuing it with the PHP internals group. If I can't get it there, I will eventually hack PHP myself or pay someone to do it.

Unfortunately for now, the only way to send anything in the body of a request, ie JSON objects or arrays, is via POST method.

So, since I don't want to waste the time to create my own message body parser for the non POST methods, (since they exist already in the POST method in PHP,) I will only use the following:

[note: if a method is in front of the URL, it is the (ACTUAL METHOD) ]

[note: this does not use 'pretty URLs'. This will be implemented later, using symfony. I will post a new update of what that looks like when I have it working. Self written code or other MVC frameworks can also implement URL rewriting/pretty URLs]

[Note: any error in any of mulitple or single objects causes all objects to be rejected, similarly to a database transaction]
=================================================

----------------------------------------------
GET, i.e. READ of C.R.U.D.
----------------------------------------------
(nothing in body of request, only in URL)
----------------------------------------------
{the collection's intial page}
....http://www.website.tld/collection-name/

{the collection's schema}
....http://www.website.tld/collection-name/?schema=TRUE

{blank object for editing and submitting as new object}
....http://www.website.tld/collection-name/?new=TRUE

{the collection paged}
....http://www.website.tld/collection-name/?start=start_id

{the collection search function}
....http://www.website.tld/collection-name/search/parameter/value/parameter/value etc

{a single JSON entity}
....http://www.website.tld/collection-name/id/?id=xxx





---------------------------------------------
POST, i.e. Create of C.R.U.D. (parameters only in body)
----------------------------------------------
(No id field in URL or allowed in new objects)
----------------------------------------------
{a new object}
....http://www.website.tld/collection-name/
........a single JSON object in POST variable named "JSON[]" or "JSON"
........individual BLOB fields in POST variable named for field
............"blob-fieldA", "blob-fieldB", etc. or "blob-fieldA[]", "blob-fieldB[]"

{a set of new objects}
....http://www.website.tld/collection-name/
........a single JSON object in POST variable named "JSON[]" or "JSON",
............but the contents is a JSON array of JSON objects
........individual BLOB fields per JSON object listed as an adjacent set,
............ sets listed same order as objects
............each in POST variable array named for field
............"blob-fieldA[]", "blob-fieldB[]"

{a set of new objects}
....http://www.website.tld/collection-name/
........multiple JSON objects each in a POST variable named "JSON[]""
........individual BLOB fields per JSON object
............each in POST variable array named for field
............"blob-fieldA[]", "blob-fieldB[]"



----------------------------------------------
PUT, i.e. UPDATE of C.R.U.D (also know as edit)
----------------------------------------------
(parameters only in body, except id field is required
as GET variable in URL for single edited object)
[NOTE: partial updates allowed. Only fields present in
submitted object will be changed. To set a field
capable of being NULL to NULL, set field in
submitted object equal to NULL, no quotes]
----------------------------------------------
{a single edited object}
......(POST)http://www.website.tld/collection-name/id/?id=xxx&_method=PUT
........a single JSON object in POST variable named "JSON[]" or "JSON"
........individual BLOB fields (as desired) in POST variable named for field
............"blob-fieldA", "blob-fieldB", etc. or "blob-fieldA[]", "blob-fieldB[]"

{a set of edited objects}
....(POST)http://www.website.tld/collection-name/?_method=PUT
........a single JSON object in POST variable named "JSON[]" or "JSON",
............but the contents is a JSON array of JSON objects
........individual BLOB fields per JSON object listed as an adjacent set,
............ sets listed same order as objects
............each in POST variable array named for field
............"blob-fieldA[]", "blob-fieldB[]"

{a set of new objects}
....http://www.website.tld/collection-name/?_method=PUT
........multiple JSON objects each in a POST variable named "JSON[]""
........individual BLOB fields per JSON object
............each in POST variable array named for field
............"blob-fieldA[]", "blob-fieldB[]"



----------------------------------------------
DELETE, i.e. DELETE of C.R.U.D
----------------------------------------------
(parameters only in body, except id field is required
as GET variable in URL for single deleted object)
----------------------------------------------
{a single deleted object}
......(DELETE)http://www.website.tld/collection-name/id/?id=xxx

{a single deleted object}
....(POST)http://www.website.tld/collection-name/id/?id=xxx&_method=DELETE

{a set of deleted objects}
....(POST)http://www.website.tld/collection-name/?_method=PUT
........a single JSON object in POST variable named "JSON[]" or "JSON",
............but the contents is a JSON array of JSON Object ids

2nd of 4 slides

This is the GETting of JSON objects which contain BLOBS using REST. It's really no different than viewing a page that contains a file server cached, remote file to unload the main server that you might visit. Think two things:

Youtube video objects on any web page that has them.
Pornography video objects.

An older more familiar application is web acceleration by Akamai.com. Remember your browser "Waiting for http://www.akamai.com/....."

1st of 4 'Slides' showing REST with BLOBS

As a primer, BLOBS ( Binary Large OBjects ) are anything that is large, and binary. Binary means that it will contain bytes with values of 0x00, and non printing control characters. This messes up string processing software everywhere in the chain of sending anything on the internet unless base32, base64, or mulitipart/mine border transmitted. Types of files that qualify are: images, videos, executables, other system files, 'binary' data files, database backups, and others.

As I have discussed previously, I did not want to send BLOBS using base64 encoding to or from a JSON based server in my current project. They base64 strings take time to encode/decode and use more bandwidth in transmission (the real bottleneck in a web application). base64 encoded BLOBS may be the 'correct' and designed way (per email with the JSON RFC author), but it's not the way I want to do it. So I borrowed from Amazon and others on the web and decided to do a 'Hybrid Server'. Look up the current web statistics and you will see that the Russian made 'nginx' (Engine X) server is taking over from Light HTTP server. The google found links that I found suggest that it is much faster and the 'wave of the future'.

Also, the PUT method of HTTP protocol (used for Update of JSON objects ) does not have any support for multipart encoding, so sending or receiving BLOBS using the PUT method from the user/app and received at the server is a laborious coding issue. Probably slow too, since I code in PHP and don't want to dig into C. My partner's app on the IPhone is in Objective C, so presumably it could be done efficiently on that platform, but why spend the time to do that?

A Hybrid server is a single domain (in this project) or even subdomain that does different fuctions on different machines. In this case, the regular php applications will be running on a standard (but optimised) LAPP machine (Linux/Apache/Postgres/PHP). Instead of storing the BLOB files in string or binary format in the Postgres Database, and requiring all four elements of the machine to process and server the large files, they will be put onto a machine running 'nginx'. The files will be permssion protected by a very simple php script running against the same database as the regular apache machine. Only permissions will be run there. The actual servering will be done by piping files directly to the user out of the filesystem.

One extra possiblity, would be live (then cached) translation of the files. If a file was stored as a video.mov, it could be requested as a video.avi, for example.

This posting on this blog contains the first of four 'slides' on the logic of the protocol. The permission system is left for a later posting. Probably it will be digest authentication. The succeding posts will probably have smaller explanations, each with one 'slide'. The slides will ge in order of 'CRUD'<---->POST/GET/PUT/DELETE. So here is POST:

PS on dual server REST server

When I can, I will submit a simple line drawing and text description of the transaction on the dual server REST blob application.