gettimeofday() issues

Get time of day returns the NUMBER of microseconds, not the digits of microseconds (6 of them) past the decimal. So to make it work with BC math, the microseconds needed to be sprintf'd in order to normalize them to 6 digits right justified. Here is the new code for both functions. Now tested for seconds rollover and 32/64bit platforms:

function make_comb_uuid(){
uuid_create(&$v4);
uuid_make($v4, UUID_MAKE_V4);
uuid_export($v4, UUID_FMT_STR, &$v4String);
$var=gettimeofday(FALSE);
return substr($v4String,0,24).substr(bcdechex($var['sec'].
 sprintf("%06d", $var['usec'])),0,12);
}

function bcdechex($dec) {
if(PHP_INT_SIZE > 16){
return dechex($dec);
} else {
$last = bcmod($dec, 16);
$remain = bcdiv(bcsub($dec, $last), 16);

if($remain == 0) {
 return dechex($last);
 } else {
   return bcdechex($remain).dechex($last);
 }
}
}

Problems with com_uuid functionn on 32 bit systems

The previous post that I wrote was tested on a 64 bit system. It was good,I swear it!

Then I uploaded it to a m1.small amazon server and the right hex digits in the output of the comb_UUID function stuck at 7fffffff. Ahhhh, the joys are overflowing ;-)

So, I got onto the PHP site and found a great little piece of code for arbitrary length DecimalToHex and HexToDecimal code.




Here is what's necessary to make this work. If you wanted to get real fancy, and I will soon, a test for size of integer should be done inside the bcdec2hex program to avoid bcmath functions.

Found a good note on the PHP site. This does it either on 32 or 64 bit system:

(credit for the guy who did it):http://www.php.net/manual/en/ref.bc.php#99130

function bcdechex($dec) {
  $last = bcmod($dec, 16);
  $remain = bcdiv(bcsub($dec, $last), 16);

  if($remain == 0) return dechex($last);
  else return bcdechex($remain).dechex($last);
}

function make_comb_uuid(){
  uuid_create(&$v4);
  uuid_make($v4, UUID_MAKE_V4);
  uuid_export($v4, UUID_FMT_STR, &$v4String);
  $var=gettimeofday(FALSE);
  return substr($v4String,0,24).substr(bcdechex($var['sec'].$var['usec']),0,12);
}

Using UUIDs for Primary Keys

There are advantages to using UUIDs for primary keys. Look at Amazon and others; They are doinng so. When exposing database ids in URLs, t allows hiding of:



A/ The number of uesrs, items, types etc.
B/ The rate of GROWTH of your site's users, items, types, etc.
C/ The SEQUENCE of ids your sites' user, item and type ids. This slows down the ability to scrape your site's data, especially using your API if you have one.



But UUID/GUID are fairly random, when made right, for certatin types. See the WikiPedia article on them at: http://en.wikipedia.org/wiki/Universally_unique_identifier (ALSO DONATE TO WIKIPEDIA TO KEEP THEM AROUND).. There are 5 types of them, and the FREE specifications are in RFC see Use a search engine and look up "UUID RFC". I believe that it is RFC4122: http://tools.ietf.org/html/rfc4122. That is not one of the easier RFCs to read,(but not as bad as the iCal ones either ;-)



Version 4 is what is needed for the most randomness. Randomness does the best job of preventing guessing and scraping of ids. However, it plays havoc with doing large abount of inserts into databases for new records, since the index pages for the primary key will be randomly accessed also. This works the database REALLY hard. it can be THIRTY times slower to use UUIDs than 4 byte, 32 bit unsigned integer primary, surrogate keys.

See the discussion of that here, with performance tests:

http://www.informit.com/articles/printerfriendly.aspx?p=25862

Since most indexes use the lowest bits of a column vlaue for hashing into index tables, if the lowest bits are close in value, or the same for a number of sequential operations, then the same index table page will remain in memory, virtually erasing any issues with using UUIDs. The author of the previous link came up with a database function that would do that, by substituting the time in microseconds, HEX based, for the last 12 characters of the UUID. The remaining 23 or so characters provide the randomness.

Since I use PHP for most of my work, I needed a PHP function that would do that. The following works well. On my 64 bit, 2.4 GhHz, 4 core, Ubuntu machine at home, just creating these values, writing it to a variable, and then creating a new one and writing it into the same variable, 1x10^06 times only took on average of 3 milliseconds TOTAL for all MILLION. So it's not much of a penalty :-) Even using SSD (Solid State Discs) for the database, that amount of delay for 1 MILLION would be below the noise level.

Here is the code:

/* requires installation of ubuntu 'php5_uuid' module
* see this URL: http://www.informit.com/articles/printerfriendly.aspx?p=25862
* else, use a search engine and look up "comb uuid", or "sequential uuid"
*
* returns a guid with the last 12 chars representing the HEX value of time
* allow better clustering of database index pages and faster performance
* while still using UUIDs
*
*/
function make_comb_uuid(){
uuid_create(&$v4);
uuid_make($v4, UUID_MAKE_V4);
uuid_export($v4, UUID_FMT_STR, &$v4String);
$var=gettimeofday();
return substr($v4String,0,24).substr(dechex($var['sec'].$var['usec']),0,12);

Printing RFCs

I am on the Linux/Ubuntu platform. 'gedit', 'firefox' (both *.txt and *.html formats) wouldn't print RFCs in a way that put a page of contents on a page of print.

RFCs are set up for dot matrix printer days.

BUT, OpenOffice 'Writer' (and probably Microsoft Word) respect the embedded ASCII page separator and properly print the document. It even worked out of the box with the default (or that chosen for *.txt formats) font type and size.

So if you need a few pages out of an RFC:
Open it in the *.txt format from the IETF website,
Save it to the file system,
Open it in OpenOffice Writer
Open 'Print Preview' and explore to find the pages you want,
Go back to 'Print', slect the pages you want,
Print them!

Testing Symfony Apps With PHPUnit

[Alt-Title: Or It's NEVER as Easy As the Tutorials Show]

Symfony development cycle is done on a local machine, (probably many other frameworks also). This saves the long upload times, and allows a programmer/developer to work on a project even while on vacation without Internet . . . . uh, right.) Is is much faster. Another major benefit is not interfering with the production site . . . at all, even in subdomain, or a separate database.

Part of that cycle is to drop and recreate the database for every little change, especially when doing testing. It means a clean slate, no interference between tests.

Symfony2 will be swtiching from its own home brewed 'lime' testing framework to PHPUnit according to Dustin Whittle as of a presentation by him at a PHP Meetup 1 week ago. They WILL be extending it to recover what they will lose from lime, gaining both from in house efforts and the community efforts @ the PHP Unit group.

So even though I'm currently using Symfony 1.4.4, I thought I would start my transition to Doctrine 2 by using PHP Unit on my current project (the subject of this blog). Both Symfony 1.4.1 and PHP Unit are current products, and I thought that I wouldn't have to deal with 'first adopter' problems. Sigh, wishful thinking.

Let me preface by saying that I prefer and use Postgresql for the database. It may not have anything to do with the lesson learned and described here, but then maybe it does. I'll leave it for the audience to comment upon using their experiences.

So, running Ubuntu, Apache, Postgres, PHP, Symfony, and Doctrine, I built a moderately complex ERD/Schema. The first lesson?

(LESSON-1)
ALWAYS use substitute, BIG INT, primary keys. Why?

• They are small compared to strings.
• If the data changes, the indexes don't have to, (data as primary keys forces the database to recalculate indexes if the data changes.)
• Symfony 1.x.x, and even Symfony 2/Doctrine 2 (according to a page at http://www.doctrine-project.com/ that I read) will not yet support a sequence with a muliti column, composite key. Just using a muliti column key would probably make the Doctrine ORM work harder and give it more chances to not do exactly what you had in mind
  

Using a substitute primary key, means that uniqueness on the DATA column potentially used as a primary key must then be enforced using an additional index on the/those column/s. So maybe the extra writing to the index still goes on. My bad. If you really want to learn about the 'Big Fight' surrounding substitute integer primary keys, use your favorite search engine to read about it. But it REALLY is better with web frameworks, trust me.

Having got past the Symfony/Doctrine framework not being able to create a model on composite keys involving a sequence, I have made great progress. In fact, I'm to the testing point on the main datagram/ERD/Schema in relationship to user input, searching, and output pages. Yehaww.

So, back to PHP Unit/Symfony. I set up an inheriting class of the main PHP Unit class 'PHPUnit_Framework_TestCase'. All methods that begin with 'test...' are executed when the class is executed by a CLI executable of PHPUnit, or when the execute() method of the class is called. They SEEM to be in the order that they are defined in the class, but I've only written two methods so far.

[LESSON-2]
Use exception test everywhere in PHPUnit derived class while doing PHPUnit/Symfony testing.
[LESSON-3]
Build fixtures (preset conditions) and tests for them INCREMENTALLY.

One reason is, the line numbers that PHPUnit feeds forward to the screen with errors is the line IT called, not where the error is. And the error stack is not that accurate either. And incremental building/testing (should be done together) lets you know, MOST of the time, where the error is.

Other reasons, inline echos come out of PHPUnit out of order from its message and its output. PHPUnit also seems to NOT output PHP errors, or stop at 'exit()', but to keep going past them. So it's better to build small pieces, build small steps,blah blah.

Did I do that at first? NO! That's why I can tell you it's easier doing so now :-)

NOW . . . The main lesson in this post.

[LESSON-4]
It's *NOT* a good idea to drop the database for every test. Reasons?

• It SEEMS as if the make a database code in Postgres or Doctrine returns before it's actually done. In fact, as I read this, I seem to remember having read somewhere that dropping/creating the database is NOT transactionable. I can tell you from personal experience, that is SEEEEEEMS that way for me. I would get errors about columns not existing, or creating tables would fail, or foreign key violations, or other errors that didn't make sense. The datasets I'm using are tiny, and I KNOW that there's no chance of foreign key violations, and which columns exist or don't.
• It takes a LONG time to drop a database and recreate one. Something like 7 seconds on my machine at home. If I am running 25-100 fixtures with several assertions each, that would be a long time, counting the data tests that will happen when the tests get much larger towards the end.
• It takes a long time to ADD test data if it's any size at all, but this also affects my eventual solution - partially.

What do do otherwise? Delete the data in the tables, reset sequences/autoincrementing columns, reload data. The database stay sin existence. Several things have to be taken into account though.

Drop the tables in the order that it does not cause any foreign key violations, from lowest 'grandchild' table to the highest parent. IF your data is too complex for that, drop all the constraints, then recreate them before repopulating the test data. If you have large datasets, and your database supports it, TRUNCATE the tables. It essentially just erases the contents of the database files but without deleting the files. Finally, if your datasets are large, or you've run many tests on the database already, force your database to scan the tables and consolidate the records to the top of the table files in a smaller file. Different DBs call it different things. If you do TRUNCATE, you may not need to do the scan.

What did I try with Symfony/Doctrine/PHPUnit *BEFORE* resorting to doing something more conventinal/pre framework era-like:

1/ Before each PHPUnit test, I ran the command line reset of the project using one of PHP's host command line invocation functions, exec(). I did it like this:

Executing it from outside of PHP while inside of the Program.

  //the following was an attempt to let Postgres drop a database.

 // Postgres will not do so while there are connections to a database.
     Doctrine_Manager::getInstance()->getCurrentConnection()->close();

exec('./symfony doctrine:build --all --and-load --no-confirmation');

This drops the database, recreates it, recreates the tables, then the constraints, then loads the data, and does not ask for confirmtatio of anything. Using that on the command line (not from within a program) is the normal command line, user land approach. This did NOT avoid any of the timing issues ,and always resulted with strange errors.

1/ Before each PHPUnit test, I ran the Symfony task from WITHIN symfony by doing this:

using a Task from inside of symfony:

  $optionsArray=array();

$argumentsArray=array();

$optionsArray[]="--all";

$optionsArray[]="--and-load";

$optionsArray[]="--no-confirmation";

$task = new sfDoctrineBuildTask($configuration->getEventDispatcher(),

new sfFormatter());

$task->run($argumentsArray, $optionsArray);

This had slightly different errors . . . . sometimes.

SOOOOOOOooooooo, since I've been in construction, I less frequently site there and try and figure out what the right thing is supposed to do, or what the designers were really trying to get us to do. I JUST GET IT WORKING. I knew that doing it the straight SQL way would work, so that's what I did.

[LESSON-5]Doctrine's 'rawSqlblah' functions aren't really 'raw' or that transparent.

BUT, I did find a post that showed how to do it, symfony framework forum: General discussion => [HOWTO] True RAW SQL in Doctrine. The thing is to avoid mixing PDO, or even lower level code withi Symfony/Docrine code because you can REALLY screw up the multi level transaction code inside of Doctrine. And, it STILL is simpler, even using raw SQL vs ORM, to use the Doctrine ORM for the connection. Your code has to be database specific in some cases, though. Here's how to do it for Postgres:

//multiple statements not possible in prepared queries (used by default)

$sql=array();

// removes every record,if the constraints allow

$sql[]="delete from table_one";

//sets it back to starting value, usually 1.

$sql[]="alter sequence table_one_id_seq restart";

// removes every record,if the constraints allow

$sql[]="delete from table_two";

//sets it back to starting value, usually 1.

$sql[]="alter sequence table_two_id_seq restart";

$doctrine = Doctrine_Manager::getInstance()->getCurrentConnection()->getDbh();

foreach( $sql as $do ){
$doctrine->query($do);
}

//Still possible, and EASIER to load the (default) fixture file using external commands:

//Symfony command line commands must always be done from project root

chdir(sfConfig::get('sf_root_dir'));
exec('./symfony doctrine:data-load');

So, this post talked about 5 lessons about using Symfony with PHPUnit that YOU don't have to learn the hard way if you don't want to:

[Alt-Title: Or It's NEVER as Easy As the Tutorials Show]
(LESSON-1)
ALWAYS use substitute, BIG INT, primary keys. Why?
[LESSON-2]
Use exception test everywhere in PHPUnit derived class while doing PHPUnit/Symfony testing.
[LESSON-3]
Build fixtures (preset conditions) and tests for them INCREMENTALLY.
[LESSON-4]
It's *NOT* a good idea to drop the database for every test.
[LESSON-5]
Doctrine's 'rawSqlblah' functions aren't really 'raw' or that transparent. See this link about the subject and better way to do it: symfony framework forum: General discussion => [HOWTO] True RAW SQL in Doctrine


Site references:
http://www.phpunit.de/
http://www.symfony-project.com/
http://www.doctrine-project.com/

[NOTES]
1/ So why was I trying to use a sequence/autoincrementing column in combination with the primary key columns? There were/are 8 integer columns that are fed from 4 other tables that need to be unique. To use THOSE as the foreign keys in the the table that is a child of that table, I really needed to make those have mulitple occurrences in that child table. BUT I wanted to be able to search for it easier and have LOTS easier database code to write, especially if I eventually go to C++ for somethings for speed. (Please, it does happen :-) There's more to it that that, proprietary to the site, that I can't divulge.

SUCCESS at doing Bulk operations in Symfony

Well, after fighting my way through many pages, and many object methods that SEEMED like they would give me the opportunity to enclose a bulk load operation in a transaction, I finally found out how to do it.

DON'T get the PDO object from the Doctrine Library. Using it directly prevents Doctrine's 'Transaction' tree from keeping track of things. The PDO object might also not be connected yet depending on your context.

A good site for better laid out, and maybe better filled out documetation on Doctrine is: http://www.tig12.net/downloads/apidocs/symfony/lib/plugins/sfDoctrinePlugin/lib/vendor/doctrine/Doctrine/Doctrine_Manager.class.html#det_methods_connection

(That's a specific link to a specific item. I'll leave it as a reader's exercise to find their way to the home page of the site ;-) )

Anyway, reading through the documentation on that site, and examples of transactions for Symfony or Doctrine, I FINALLY found the crucial details.

A/ Use an instance of Doctrine_Connection to handle transactions external to the normal ORM statements that might be done for bulk loading.

B/ HOW TO GET THAT Doctrine_Connection instance while in a connected state.

And, drum roll please,this is how you do it:

$connection=Doctrine_Manager::connection(); // Doctrine_Manager is static.
//read link above to get info on how it automatically connects

$connection->beginTransaction();

//do bulk load stuff here, orrrrrrrrr
//roll your own save around all the elements of a complex schema save.
//For example.
//Books<->Authors database
//Authors can have multiple books
//Books can have multiple authors
//Three tables/objects in database
//Books, BookAuthors, Authors.
//To get them to save all at one

$book=new Book();
$book['author']='george';
$book['author']='dude';
$book->save();

//$book now has two authors.
//Internal code to books (and/or authors) that you have to write collects all the
//authors for books, or books for authors, and saves to all three tables, in between
// 'beginTransaction()' like above, and a 'commitTransaction()', like below.

$connection->beginTransaction();

Simplistic, and effective, but not complete. To do it right, you would put transaction inside of Book->save() method inside a try/catch block to catch database exceptions. See http://www.php.net/try and read all relevant links for 'catch', 'Exception', 'throw'

How this helps you.

EXTRA TIP.

Use arrays for data input and output as much as possible to save overhead

Set all variables to NULL at end of loops to help memory management and garbage collection in PHP.

Using the database in Tasks in Symfony

The latest Symfony (1.4.1 as of 2010-06-13, using doctrine 1.2) has a pretty good task skeleton generator.

But first, the definition of a task. For Symfony, a task is something that can be executed using PHP CLI (Command Line Interpreter). This is also something that can be executed from a cron script (a good thing to know).

The key part is, that the skeleton Symfony creates uses the whole Symfony environment, including the Doctrine ORM, filters, all sorts of things. However, the skeleton, like a LOT of things in Symfony, is not well documented. Or, it's documented in only a tutorial. Hence, this blog article on the database connection.

Here is the code that my use of symfony generated:

The command line issued while in the project directory
~$ ./symfony generate:task taskman
>> task Creating "/home/project_dir/lib/ta...taskmanTask.class.php" task file
~$

The resultant file in /home/project_dir/lib/task/taskmanTask.php
<?php

class taskmanTask extends sfBaseTask
{
protected function configure()
{
// // add your own arguments here
// $this->addArguments(array(
// new sfCommandArgument('my_arg', sfCommandArgument::REQUIRED, 'My argument'),
// ));

$this->addOptions(array(
new sfCommandOption('application', null, sfCommandOption::PARAMETER_REQUIRED, 'The application name'),
new sfCommandOption('env', null, sfCommandOption::PARAMETER_REQUIRED, 'The environment', 'dev'),
new sfCommandOption('connection', null, sfCommandOption::PARAMETER_REQUIRED, 'The connection name', 'doctrine'),
// add your own options here
));

$this->namespace = '';
$this->name = 'taskman';
$this->briefDescription = '';
$this->detailedDescription = <<<EOF
The [taskman|INFO] task does things.
Call it with:

[php symfony taskman|INFO]
EOF;
}

protected function execute($arguments = array(), $options = array())
{
// initialize the database connection
$databaseManager = new sfDatabaseManager($this->configuration);
$connection = $databaseManager->getDatabase($options['connection'])->getConnection();

// add your code here
}
}
Notice these excerpted lines:
// initialize the database connection
$databaseManager = new sfDatabaseManager($this->configuration);
$connection = $databaseManager->getDatabase($options['connection'])->getConnection();

The big deal is that it happens in the context of the task class, inheriting the BaseTask class. So that's where the '$this->configuration' argument comes from. This configuration contains the database connection details for /home/project_dir/config/databases.yml, among other things.

So the variable $databaseManager, and instance of sfDatabaseManager, already has set up in it all your connections defined in databases.yml. 'You should know that', right? ;-) You should also know how the dbases in your databases are named, right? Yeah, I thought not.

It's not really apparent from the code generated byt $options['connection'] is how you feed in the name of the database connection as defined in databases.yml. Here is my simple,for now, databases.yml (mangled to remove important details):

# You can find more information about this file on the symfony website:
# http://www.symfony-project.org/reference/1_4/en/07-Databases

all:
doctrine:
class: sfDoctrineDatabase
param:
dsn: pgsql:host=localhost;dbname=dbname
username: semi_administrative_name
password: separate_password_for_each_connection_username

I don't remember how I generated this yml file, I think it was automatic as a Doctrine one upon project generation. Anyway, 'doctrine' is the name of the connection.

There are two ways you could feed this into the statement:
'WAY ONE'
$connection = $databaseManager->getDatabase('doctrine')->getConnection();

'WAY TWO'
feed the name 'doctrine' to the task on the command line as an option, like:
{from your project directory}
./symfony namespace:taskmanTask.php --connection=doctrine
$connection = $databaseManager->getDatabase($options['connection'])->getConnection();


Now, three other things to consider to round out using the databases in Symfony tasks:

A/ You can investigate using sfOrmTask as the base class for your task, like:

class taskmanTask extends sfDoctrineTask
vs
class taskmanTask extends sfBaseTask
Some hints on how that might be useful are here:
http://librosweb.es/symfony_1_2_en/capitulo16/using_symfony_outside_of_a_web_context.html

B/ You need to declare a namespace inside of the generated task file. Symfony will automatically scan all tasks and cache them and therefore will know the namespace declared inside the file. For example:

change:
$this->namespace = '';
to:
$this->namespace = 'cron'; // I use this for my cron running scripts

C/ I forgot, what it was, if I remember, I'll edit this.

Anyway, passing this, probably incomplete, info on using databases in tasks.

PS SOMEDAY I HAVE TO LEARN HOW TO KEEP INDENTED LINES IN CODE HERE. The .yml file has various indentions. See your own file.

Obivously, folks, I'm not a great schmoozer and publicist for myself. SOME of that is going to change - It's a necessary business skill.

Highlights of my absence:

1/ Currently laid off, between jobs, etc. Getting in Better Shape, getting allthe 'hanging over my head things I need to do someday' done.

2/ Some insights into using cron.
A/ On Ubuntu, the cron scripts for individual users are stored in a different place from where all the web articles said they were. Ubuntu does this often, but often better than other distributions, IMHO.

The location is: /var/spool/cron/crontabs/$SYSTEM_USER_NAME

B/ For something to happen at a repeating schedule, use */num. I saw this listed as '0/num' in several places, which is different. THAT starts at the 0 value of the time measure (minute/hour/day/month/year) and then every 'num' units past that. '*/num' starts at the NEXT evenly divisible by NUM unit. Examples:

EVERY MINUTE ----------------------------------
# m h dom mon dow command
0/1 * * * * echo "every single minute" >> $HOME/cron.log

Will create (if necessary) append (if file exists) the phrase 'every single minute' from the first time it is called by the cron daemon.

EVERY MINUTE AFTER XX:XX:00
# m h dom mon dow command
0/1 * * * * echo "every minute after the top of the next hour and then forever" >> $HOME/cron.log

Will create (if necessary) append (if file exists) the phrase 'every minute after the next top of the next hour and then forever' by the cron daemon.

GREAT service to ease introducing and monetizing an API

I highly recommend the following company's product and concept.

http://www.webservius.com/

It handles the APP_id sign up, authentication, bandwidth throttling, and many other issuses. They said that they use cloud servers so they shouldn't provide much dealy between your users and your api.

There's a free version for you and your clients.

Tell them that I sent you? I get nothing but a warm fuzzy feeling when he calls me thanking me.

Success in API development and concepts learned

Our company had a succesful introduction of our API at a 'Hackathon' in Mountain View, CA yesterday. Lot's of great ideas are brewing at the 'Hacker Dojo'. Problems that we have solved, or found solutions for are:

1/ How to indicate the format of a feed/api in a REST URL. The old standard of putting the file type extension at the end seems to be universal and easiest to implement. Examples: (format can be 'json','xml','txt','html','pdf',etc)

create new resource in collection called 'resource_name'
POST http://sub.domain.tld/resource_name.format
POST http://api.yahoo.com/calendar.json (made this up)
(returns 'resource_id')

get resource in collection called 'resource_name'
GET http://sub.domain.tld/resource_name/resource_id.format
GET http://api.yahoo.com/calendar/a8e5b892c0024ead.json (made this up)

get resources in collection called 'resource_name'
GET http://sub.domain.tld/resource_name.format?query_string
GET http://api.yahoo.com/calendar.json?search_text=danc&from_date=2010-03-01&to_date=2010-03-07 (made this up)

2/ It's an INSANE consumer of memory and speed to format outgoing JSON/XML/etc for human readability. Rely on the user's viewing software application for that. DON'T format your output data for human readability. (On a shared hosting account, for 250 records returned in JSON, it would time out @ 30 seconds while formatted while returning nothing. Taking the formatting out the started the 'transferring data from site-name' message after 1 second, and it was basically my wireless holding up that transfer which took 5 seconds for 500kbytes. There may also have been some time for the browser to render the JSON using the 'JSONView' plugin, nicely formatting it :-)

There are some other things we have learned lately. I will post them over time.

All the best out there.

Research and Experience gained

It's been a long time, my poor little blog (and it's followers(s)). A lot of my work is now going to be company confidential. But I will share what I have figured out . . . if it's already out on the web somewhere.

1) Most Server site software development frameworks now use URL rewriting.URL rewrite can 'scrape' variable/value pairs out of apparent directories after a site address. For example:

http://www.site.tld/variable-A/value-A/animal-type/dog/

The delimiters can be fairly custom within the allowed URL character set. For example:

http://www.site.tld/variable-A#value-A/animal-type#dog/
http://www.site.tld/variable-A#value-A/animal-type#dog/breeds#poodle;afghan;border-collie/

2) When using a modern software development framwork like Symfony, Ruby on Rails, .Net MVC, and the Java MVC products, one of the VERY FIRST THINGS THAT YOU WANT TO DO between doing all the business modeling, use cases, and other top level software design tasks, and actually coding is to map the resources and modules to URLs. Part of that is deciding what formats to supply upon request and where to signify the format on the URL.


3) If a site is going to use multiple formats, the most common way for that to to the server what format is desired is to use a file extension, but use it ubiquitously.

That means, as normal, if a URL looks like:

http://subdomain.domain.tld/directory/filename.ext

Just change the extension to what is desired, and if the site supports it, grand. Usual suspects are .rss|.xml|.json|.html. The ubiquitous part means NO MATTER WHAT IS THE LAST MAJOR VALUE AT THE END OF THE URL (except for variable/value pairs and query strings), PUT THE '.format' AT THE END OF IT. Examples will show it clearly:

http://subdomain.domain.tld/module-or-resource/action.FORMAT

http://subdomain.domain.tld/fake_directory_name_for_user_readability/module-or-resource/action.FORMAT

http://subdomain.domain.tld/module-or-resource/action.FORMAT/var1#value1/var2#value2/

http://subdomain.domain.tld/fake_directory_name_for_user_readability/module-or-resource/action.FORMAT?not-user-friendly-query-string-value=something&also-serach-engines-dont-catalog-this-full-url=but-we-want-that

http://www.site.tld/resource/action/id.format

http://www.site.tld/products/ship/4d912f22c182293a70e2e7ac3671228dff397a52.FORMAT/carrier#ups/rate#blue/address#123-Mocking-Bird-Lane-Uphigh-CO-80230-USA/currency#usd


The '.format' is between the end of the part of the URL that selects the resource or code and the part of the URL that supplies variables to the code processing the request. (PHP/Symfony, PHP/Drupal, Ruby/Rails, et al. all use this)

This convention/practice was decided upon in my projects after going through the research below. I hope it is as useful to you as it was to me:

=========================================

=================
I looked at the following sites:
Twitter
yahoo
amazon S3
facebook
myspace
linkedin
google.

(EOL = End of the Last listed of Collection, Module, Action ,Or Id)

The methods of specifying the return type of the document were one of three different kinds:

1/ Implied, because only one was available (Amazon S3:json, Linkedin:xml)
2/ Implied but another could be specified BY ADDING ‘.format’ TO THE END OF THE EOL before any query string or friendly URL components at the end of the whole URL (MySpace-xml:add.json)
3/ Implied but another could be specified by a query string paramter (google-atom:‘alt=json’, yahoo-xml:‘output=json’)
4/ Format was required BY ADDING ‘.format’ TO THE END OF THE EOL ((Twitter:add .json|.xml|.rss)

=========================================

API Versions numbers used in URL of the the API for:
Yahoo
MySpace

EXAMMPLES of the way '.format' is added to API URLs:
-------------------------------------------------------
http://microformats.org/wiki/rest/urls
HTML
 GET /people/1
     return the first record in HTML format
 GET /people/1.html
     return the first record in HTML format
XML
 GET /people/1.xml
     return the first record in XML format
JSON
 GET /people/1.json
     return the first record in JSON format

http://confluence.sakaiproject.org/display/SAKDEV/EntityBroker+RESTful+URL+support

Access to an entity:(MEMBER)

   * http://localhost:8080/direct/webapp-entity/id0
   * http://localhost:8080/direct/webapp-entity/id0.xml
   * http://localhost:8080/direct/webapp-entity/id0.json

Access to an entity space:(COLLECTION)

   * http://localhost:8080/direct/webapp-entity
   * http://localhost:8080/direct/webapp-entity.xml
   * http://localhost:8080/direct/webapp-entity.json

Describing entities: (SELF-DISCOVERY/ERD)
   o http://localhost:8080/direct/describe
   o http://localhost:8080/direct/eval-evaluation/describe
   o http://localhost:8080/direct/webapp-entity/describe
   o http://localhost:8080/direct/webapp-entity/describe.xml

MySpace
EXAMPLE REQUEST:
   * XML: http://api.myspace.com/v1/users/454304609/albums
   * JSON: http://api.myspace.com/v1/users/454304609/albums.json

Twitter
http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search
Search(GET)
 o Example: http://search.twitter.com/search.json?callback=foo&q=twitter
            http://search.twitter.com/search.atom?lang=en&q=devo
Statuses/User_timeline

http://twitter.com/statuses/user_timeline/12345.xml or http://twitter.com/statuses/user_timeline/bob.json.
http://twitter.com/statuses/user_timeline.xml?user_id=1401881
http://twitter.com/statuses/user_timeline.xml?screen_name=101010
http://twitter.com/statuses/user_timeline.rss?page=3
Status Updates (POST)
curl -u user:password -d "status=playing with cURL and the Twitter API" http://twitter.com/statuses/update.xml


 
Further References
http://www.xml.com/pub/a/2004/12/01/restful-web.html
http://ajaxpatterns.org/RESTful_Service (EXCELLENT)*  Describing entities:
   o http://localhost:8080/direct/describe
   o http://localhost:8080/direct/eval-evaluation/describe
   o http://localhost:8080/direct/webapp-entity/describe
   o http://localhost:8080/direct/webapp-entity/describe.xml

Symfony recognizes four main http methods

Well, it turns out that Symfony gets loaded in all the methods except HEAD, as expected.

I tried:
-----------
GET
POST
DELETE
PUT
HEAD
OPTIONS

Symfony reported the method correctly in the ones that really matter, the first 4 above. In Options, it displayed GET as the method. In HEAD, it was never invoked, which agrees with the HTTP specification, so that's a good thing.

Well, that DOESN'T change the fact that PHP won't parse the body for DELETE or PUT requests and so there is no pretty access to the body. I'd have to come up with my own MIME/POST processor. Not going there :-(

So, I will use the POST for all PUTS, and probably most multiple DELETES. As I had originally talked about in a previous blog.

However, in Symfony, I need to deal with the crsf variable in the body of the request, and NO direct GET variables in the URL. Mr. Ponticier (creator of Symfony) has added an optional,standard POST/body-of-the-request variable for 'method'. I may or may not use that. Investigating now.

PS., one other thing that I would have to deal with is Symfony's labeling of variables, they already use the brackets for their use of an array of values for columns of a table.

And I tested once, PHP doesn't accept multiple sets of brackets for arrays of arrays in the names of POST variables.
Investigating.

One step back

Well, I had the code for doing a REST-POST(single or arrayed) as I have previously described completely done. Then I learned what SVN and backups should be good for :-(

I found out a way to switch two files, do my normal neurotic 1 save/min, and end up with only one of two files that I had written. Since I had been working on it 4-6 hours a day, in addition to my full time job, I was exhausted and despondent after doing it.

So, I stepped back, relaxed, and decided to skip the temporary functional style design and go straight to Symfony. I am reading all I can on REST a la Symfony. I don't totally agree with what I've seen, and especially, from the founder of Symfony. However, he is much better at using his framework than I am ;-) So I will keep an open mind as I try to do it, 'my way'. (see previous posts on this blog)

I am just about to see start a micro project to test two things:
1/ How to use the crsf token that Symfony generates in a REST environment.
2/ If the sfWebRequest object contains the Method used to access the script, and see if I can actually at least use single JSON DELETE/POST.

Someday, I am going to pay someone to hack PHP and allow the $_POST variables to be filled during POST/DELETE/PUT/GET methods.

Implementing JSON REST with BLOBs in PHP - The practical way.

I have retrenched, somewhat, (if my memory of what I've been writing serves me correctly.)

I have decided to skip the use of the PUT method until PHP supports it better.

I would like to see full $_GET, $_POST, $_PUT, $_DELETE, $_COOKIE, $_OPTION, $_HEAD, $_REQUEST, etc request body support for the appropriate methods in PHP. From what I've read, this is within the full specs of the HTTP protocol. I'm pursuing it with the PHP internals group. If I can't get it there, I will eventually hack PHP myself or pay someone to do it.

Unfortunately for now, the only way to send anything in the body of a request, ie JSON objects or arrays, is via POST method.

So, since I don't want to waste the time to create my own message body parser for the non POST methods, (since they exist already in the POST method in PHP,) I will only use the following:

[note: if a method is in front of the URL, it is the (ACTUAL METHOD) ]

[note: this does not use 'pretty URLs'. This will be implemented later, using symfony. I will post a new update of what that looks like when I have it working. Self written code or other MVC frameworks can also implement URL rewriting/pretty URLs]

[Note: any error in any of mulitple or single objects causes all objects to be rejected, similarly to a database transaction]
=================================================

----------------------------------------------
GET, i.e. READ of C.R.U.D.
----------------------------------------------
(nothing in body of request, only in URL)
----------------------------------------------
{the collection's intial page}
....http://www.website.tld/collection-name/

{the collection's schema}
....http://www.website.tld/collection-name/?schema=TRUE

{blank object for editing and submitting as new object}
....http://www.website.tld/collection-name/?new=TRUE

{the collection paged}
....http://www.website.tld/collection-name/?start=start_id

{the collection search function}
....http://www.website.tld/collection-name/search/parameter/value/parameter/value etc

{a single JSON entity}
....http://www.website.tld/collection-name/id/?id=xxx





---------------------------------------------
POST, i.e. Create of C.R.U.D. (parameters only in body)
----------------------------------------------
(No id field in URL or allowed in new objects)
----------------------------------------------
{a new object}
....http://www.website.tld/collection-name/
........a single JSON object in POST variable named "JSON[]" or "JSON"
........individual BLOB fields in POST variable named for field
............"blob-fieldA", "blob-fieldB", etc. or "blob-fieldA[]", "blob-fieldB[]"

{a set of new objects}
....http://www.website.tld/collection-name/
........a single JSON object in POST variable named "JSON[]" or "JSON",
............but the contents is a JSON array of JSON objects
........individual BLOB fields per JSON object listed as an adjacent set,
............ sets listed same order as objects
............each in POST variable array named for field
............"blob-fieldA[]", "blob-fieldB[]"

{a set of new objects}
....http://www.website.tld/collection-name/
........multiple JSON objects each in a POST variable named "JSON[]""
........individual BLOB fields per JSON object
............each in POST variable array named for field
............"blob-fieldA[]", "blob-fieldB[]"



----------------------------------------------
PUT, i.e. UPDATE of C.R.U.D (also know as edit)
----------------------------------------------
(parameters only in body, except id field is required
as GET variable in URL for single edited object)
[NOTE: partial updates allowed. Only fields present in
submitted object will be changed. To set a field
capable of being NULL to NULL, set field in
submitted object equal to NULL, no quotes]
----------------------------------------------
{a single edited object}
......(POST)http://www.website.tld/collection-name/id/?id=xxx&_method=PUT
........a single JSON object in POST variable named "JSON[]" or "JSON"
........individual BLOB fields (as desired) in POST variable named for field
............"blob-fieldA", "blob-fieldB", etc. or "blob-fieldA[]", "blob-fieldB[]"

{a set of edited objects}
....(POST)http://www.website.tld/collection-name/?_method=PUT
........a single JSON object in POST variable named "JSON[]" or "JSON",
............but the contents is a JSON array of JSON objects
........individual BLOB fields per JSON object listed as an adjacent set,
............ sets listed same order as objects
............each in POST variable array named for field
............"blob-fieldA[]", "blob-fieldB[]"

{a set of new objects}
....http://www.website.tld/collection-name/?_method=PUT
........multiple JSON objects each in a POST variable named "JSON[]""
........individual BLOB fields per JSON object
............each in POST variable array named for field
............"blob-fieldA[]", "blob-fieldB[]"



----------------------------------------------
DELETE, i.e. DELETE of C.R.U.D
----------------------------------------------
(parameters only in body, except id field is required
as GET variable in URL for single deleted object)
----------------------------------------------
{a single deleted object}
......(DELETE)http://www.website.tld/collection-name/id/?id=xxx

{a single deleted object}
....(POST)http://www.website.tld/collection-name/id/?id=xxx&_method=DELETE

{a set of deleted objects}
....(POST)http://www.website.tld/collection-name/?_method=PUT
........a single JSON object in POST variable named "JSON[]" or "JSON",
............but the contents is a JSON array of JSON Object ids

2nd of 4 slides

This is the GETting of JSON objects which contain BLOBS using REST. It's really no different than viewing a page that contains a file server cached, remote file to unload the main server that you might visit. Think two things:

Youtube video objects on any web page that has them.
Pornography video objects.

An older more familiar application is web acceleration by Akamai.com. Remember your browser "Waiting for http://www.akamai.com/....."

1st of 4 'Slides' showing REST with BLOBS

As a primer, BLOBS ( Binary Large OBjects ) are anything that is large, and binary. Binary means that it will contain bytes with values of 0x00, and non printing control characters. This messes up string processing software everywhere in the chain of sending anything on the internet unless base32, base64, or mulitipart/mine border transmitted. Types of files that qualify are: images, videos, executables, other system files, 'binary' data files, database backups, and others.

As I have discussed previously, I did not want to send BLOBS using base64 encoding to or from a JSON based server in my current project. They base64 strings take time to encode/decode and use more bandwidth in transmission (the real bottleneck in a web application). base64 encoded BLOBS may be the 'correct' and designed way (per email with the JSON RFC author), but it's not the way I want to do it. So I borrowed from Amazon and others on the web and decided to do a 'Hybrid Server'. Look up the current web statistics and you will see that the Russian made 'nginx' (Engine X) server is taking over from Light HTTP server. The google found links that I found suggest that it is much faster and the 'wave of the future'.

Also, the PUT method of HTTP protocol (used for Update of JSON objects ) does not have any support for multipart encoding, so sending or receiving BLOBS using the PUT method from the user/app and received at the server is a laborious coding issue. Probably slow too, since I code in PHP and don't want to dig into C. My partner's app on the IPhone is in Objective C, so presumably it could be done efficiently on that platform, but why spend the time to do that?

A Hybrid server is a single domain (in this project) or even subdomain that does different fuctions on different machines. In this case, the regular php applications will be running on a standard (but optimised) LAPP machine (Linux/Apache/Postgres/PHP). Instead of storing the BLOB files in string or binary format in the Postgres Database, and requiring all four elements of the machine to process and server the large files, they will be put onto a machine running 'nginx'. The files will be permssion protected by a very simple php script running against the same database as the regular apache machine. Only permissions will be run there. The actual servering will be done by piping files directly to the user out of the filesystem.

One extra possiblity, would be live (then cached) translation of the files. If a file was stored as a video.mov, it could be requested as a video.avi, for example.

This posting on this blog contains the first of four 'slides' on the logic of the protocol. The permission system is left for a later posting. Probably it will be digest authentication. The succeding posts will probably have smaller explanations, each with one 'slide'. The slides will ge in order of 'CRUD'<---->POST/GET/PUT/DELETE. So here is POST:

PS on dual server REST server

When I can, I will submit a simple line drawing and text description of the transaction on the dual server REST blob application.

Discorvery/Reinventing the Wheel

After much research, I came to the conclusion that JSON is great for
what I want to do except for one thing - binary images. (This research also showed the reason why there are so many 'partial' adoptions of JSON.)

Binary files as fields in JSON objects are especially a problem during a PUT operations. In all the server script languages and in Apache itself, there is access to GET and POST parameters. There is no such mechanism in PUT. One has to write a script to create the boundaries for mulitpart PUTted files in your own application, AND in script for manually dividing up the body of the put based on the boundaries to claim them on the SERVER. In scripting languages, this is very slow.

Of course, it could be sent as base64 encoded strings, which is what
the designer of JSON had in mind, but that is EVEN SLOWER than scanning for boundary markers in a POST style body using a script language. (These are all assumptions on my part, not testing done.)

So my approach is simliar to Amazon's S3 storage server:
http://www.anyexample.com/programming/php/uploading_files_to_amazon_s... and to Yahoo's email attachment option, (UPlaid files for storage and virus scan, then send body of email)

I will have the user POST binary files to a separate SUB domain, using Apache-LIght (a seperate process on a seperate, very fast server), and the server will return a 'bucket', or random file name, (protected by permission headers in between client and host , i.e, header cookies).

The application on a browser or other (IPhone non browser app for
example) Will then populate the fileds of the JSON with the returned
server/filename, as well as the usual binary file related stuff, i.e. title, type, etc.

Applications GETting JSON objects, or PUTting Objects will get or send binary files the same way.

And also, I don't like the semantics of PUTting to originate a JSON
object. So I will be restricting PUT to the Update only of standard
CRUD, (Create, Read, Update, Delete). I don't want users setting
primary key fields.

So, I hope this saves other people some effort that I went through.

The next post, sometime today, will explain what I found out about PUT/DELETE methods in HTTP on Apache @ A2Hosting (I use them because of good support and Postgres databases). To properly use REST, and not use kludgy URLs using both POST and GET values at the same time, (totally doable, but not clean), PUT and DELETE are necessary complements to the standard POST and GET methods.

Well, a little history on me and blogs. I am an older person, so I am more conservative. I am also the first born. So for me, public self expression is only good for dancing :-) Giving the world my opinion and being 'an authority', in public, is not emotionally comfortable. (Giving unwanted advice to strangers on planes, busses, trains is not a problem. Remember, I am a first born ;-)

So, I haven't read too many blogs unless Google found them for me. I don't follow but one, inactive blog. I don't look for an audience for my blog. So .... I'll answer the obvious question.

If I find something that takes me a long time to figure out, I will post the answer here for everyone else's benefit - (Usually in the Open Source community.) I owe the Open Source world a lot, we all do. The web would be a mish mash of standards, most by Microslop, if Open Source were not the dominant force there. ALL your interactions with the web would be mostly to benefit Microslop's marketing focus.

I am having a VERY difficult time finding clear material on how POST and DELETE methods are used on Apache HTTP servers. All the 'great' RESTful articles discuss that HTTP servers can do it, and 'just set it up' to take care of the 'U' and 'D' of CRUD (Create-Read-Update-Delete) of basic content management, AKA, REST.

My next blog post on this blog will discuss what I have found so far, and the details I have not worked out yet. Eventually, this blog 'subject'(?) will show exactly how it is done and the pitfalls found trying to 'make it so'.

This is my first blog. Been working on how to do BLOBs in REST on php and keeping it totally internationized (I18N). So expect more during the next week.