Using the database in Tasks in Symfony

The latest Symfony (1.4.1 as of 2010-06-13, using doctrine 1.2) has a pretty good task skeleton generator.

But first, the definition of a task. For Symfony, a task is something that can be executed using PHP CLI (Command Line Interpreter). This is also something that can be executed from a cron script (a good thing to know).

The key part is, that the skeleton Symfony creates uses the whole Symfony environment, including the Doctrine ORM, filters, all sorts of things. However, the skeleton, like a LOT of things in Symfony, is not well documented. Or, it's documented in only a tutorial. Hence, this blog article on the database connection.

Here is the code that my use of symfony generated:

The command line issued while in the project directory
~$ ./symfony generate:task taskman
>> task Creating "/home/project_dir/lib/ta...taskmanTask.class.php" task file
~$

The resultant file in /home/project_dir/lib/task/taskmanTask.php
<?php

class taskmanTask extends sfBaseTask
{
protected function configure()
{
// // add your own arguments here
// $this->addArguments(array(
// new sfCommandArgument('my_arg', sfCommandArgument::REQUIRED, 'My argument'),
// ));

$this->addOptions(array(
new sfCommandOption('application', null, sfCommandOption::PARAMETER_REQUIRED, 'The application name'),
new sfCommandOption('env', null, sfCommandOption::PARAMETER_REQUIRED, 'The environment', 'dev'),
new sfCommandOption('connection', null, sfCommandOption::PARAMETER_REQUIRED, 'The connection name', 'doctrine'),
// add your own options here
));

$this->namespace = '';
$this->name = 'taskman';
$this->briefDescription = '';
$this->detailedDescription = <<<EOF
The [taskman|INFO] task does things.
Call it with:

[php symfony taskman|INFO]
EOF;
}

protected function execute($arguments = array(), $options = array())
{
// initialize the database connection
$databaseManager = new sfDatabaseManager($this->configuration);
$connection = $databaseManager->getDatabase($options['connection'])->getConnection();

// add your code here
}
}
Notice these excerpted lines:
// initialize the database connection
$databaseManager = new sfDatabaseManager($this->configuration);
$connection = $databaseManager->getDatabase($options['connection'])->getConnection();

The big deal is that it happens in the context of the task class, inheriting the BaseTask class. So that's where the '$this->configuration' argument comes from. This configuration contains the database connection details for /home/project_dir/config/databases.yml, among other things.

So the variable $databaseManager, and instance of sfDatabaseManager, already has set up in it all your connections defined in databases.yml. 'You should know that', right? ;-) You should also know how the dbases in your databases are named, right? Yeah, I thought not.

It's not really apparent from the code generated byt $options['connection'] is how you feed in the name of the database connection as defined in databases.yml. Here is my simple,for now, databases.yml (mangled to remove important details):

# You can find more information about this file on the symfony website:
# http://www.symfony-project.org/reference/1_4/en/07-Databases

all:
doctrine:
class: sfDoctrineDatabase
param:
dsn: pgsql:host=localhost;dbname=dbname
username: semi_administrative_name
password: separate_password_for_each_connection_username

I don't remember how I generated this yml file, I think it was automatic as a Doctrine one upon project generation. Anyway, 'doctrine' is the name of the connection.

There are two ways you could feed this into the statement:
'WAY ONE'
$connection = $databaseManager->getDatabase('doctrine')->getConnection();

'WAY TWO'
feed the name 'doctrine' to the task on the command line as an option, like:
{from your project directory}
./symfony namespace:taskmanTask.php --connection=doctrine
$connection = $databaseManager->getDatabase($options['connection'])->getConnection();


Now, three other things to consider to round out using the databases in Symfony tasks:

A/ You can investigate using sfOrmTask as the base class for your task, like:

class taskmanTask extends sfDoctrineTask
vs
class taskmanTask extends sfBaseTask
Some hints on how that might be useful are here:
http://librosweb.es/symfony_1_2_en/capitulo16/using_symfony_outside_of_a_web_context.html

B/ You need to declare a namespace inside of the generated task file. Symfony will automatically scan all tasks and cache them and therefore will know the namespace declared inside the file. For example:

change:
$this->namespace = '';
to:
$this->namespace = 'cron'; // I use this for my cron running scripts

C/ I forgot, what it was, if I remember, I'll edit this.

Anyway, passing this, probably incomplete, info on using databases in tasks.

PS SOMEDAY I HAVE TO LEARN HOW TO KEEP INDENTED LINES IN CODE HERE. The .yml file has various indentions. See your own file.

Obivously, folks, I'm not a great schmoozer and publicist for myself. SOME of that is going to change - It's a necessary business skill.

Highlights of my absence:

1/ Currently laid off, between jobs, etc. Getting in Better Shape, getting allthe 'hanging over my head things I need to do someday' done.

2/ Some insights into using cron.
A/ On Ubuntu, the cron scripts for individual users are stored in a different place from where all the web articles said they were. Ubuntu does this often, but often better than other distributions, IMHO.

The location is: /var/spool/cron/crontabs/$SYSTEM_USER_NAME

B/ For something to happen at a repeating schedule, use */num. I saw this listed as '0/num' in several places, which is different. THAT starts at the 0 value of the time measure (minute/hour/day/month/year) and then every 'num' units past that. '*/num' starts at the NEXT evenly divisible by NUM unit. Examples:

EVERY MINUTE ----------------------------------
# m h dom mon dow command
0/1 * * * * echo "every single minute" >> $HOME/cron.log

Will create (if necessary) append (if file exists) the phrase 'every single minute' from the first time it is called by the cron daemon.

EVERY MINUTE AFTER XX:XX:00
# m h dom mon dow command
0/1 * * * * echo "every minute after the top of the next hour and then forever" >> $HOME/cron.log

Will create (if necessary) append (if file exists) the phrase 'every minute after the next top of the next hour and then forever' by the cron daemon.

GREAT service to ease introducing and monetizing an API

I highly recommend the following company's product and concept.

http://www.webservius.com/

It handles the APP_id sign up, authentication, bandwidth throttling, and many other issuses. They said that they use cloud servers so they shouldn't provide much dealy between your users and your api.

There's a free version for you and your clients.

Tell them that I sent you? I get nothing but a warm fuzzy feeling when he calls me thanking me.

Success in API development and concepts learned

Our company had a succesful introduction of our API at a 'Hackathon' in Mountain View, CA yesterday. Lot's of great ideas are brewing at the 'Hacker Dojo'. Problems that we have solved, or found solutions for are:

1/ How to indicate the format of a feed/api in a REST URL. The old standard of putting the file type extension at the end seems to be universal and easiest to implement. Examples: (format can be 'json','xml','txt','html','pdf',etc)

create new resource in collection called 'resource_name'
POST http://sub.domain.tld/resource_name.format
POST http://api.yahoo.com/calendar.json (made this up)
(returns 'resource_id')

get resource in collection called 'resource_name'
GET http://sub.domain.tld/resource_name/resource_id.format
GET http://api.yahoo.com/calendar/a8e5b892c0024ead.json (made this up)

get resources in collection called 'resource_name'
GET http://sub.domain.tld/resource_name.format?query_string
GET http://api.yahoo.com/calendar.json?search_text=danc&from_date=2010-03-01&to_date=2010-03-07 (made this up)

2/ It's an INSANE consumer of memory and speed to format outgoing JSON/XML/etc for human readability. Rely on the user's viewing software application for that. DON'T format your output data for human readability. (On a shared hosting account, for 250 records returned in JSON, it would time out @ 30 seconds while formatted while returning nothing. Taking the formatting out the started the 'transferring data from site-name' message after 1 second, and it was basically my wireless holding up that transfer which took 5 seconds for 500kbytes. There may also have been some time for the browser to render the JSON using the 'JSONView' plugin, nicely formatting it :-)

There are some other things we have learned lately. I will post them over time.

All the best out there.

Research and Experience gained

It's been a long time, my poor little blog (and it's followers(s)). A lot of my work is now going to be company confidential. But I will share what I have figured out . . . if it's already out on the web somewhere.

1) Most Server site software development frameworks now use URL rewriting.URL rewrite can 'scrape' variable/value pairs out of apparent directories after a site address. For example:

http://www.site.tld/variable-A/value-A/animal-type/dog/

The delimiters can be fairly custom within the allowed URL character set. For example:

http://www.site.tld/variable-A#value-A/animal-type#dog/
http://www.site.tld/variable-A#value-A/animal-type#dog/breeds#poodle;afghan;border-collie/

2) When using a modern software development framwork like Symfony, Ruby on Rails, .Net MVC, and the Java MVC products, one of the VERY FIRST THINGS THAT YOU WANT TO DO between doing all the business modeling, use cases, and other top level software design tasks, and actually coding is to map the resources and modules to URLs. Part of that is deciding what formats to supply upon request and where to signify the format on the URL.


3) If a site is going to use multiple formats, the most common way for that to to the server what format is desired is to use a file extension, but use it ubiquitously.

That means, as normal, if a URL looks like:

http://subdomain.domain.tld/directory/filename.ext

Just change the extension to what is desired, and if the site supports it, grand. Usual suspects are .rss|.xml|.json|.html. The ubiquitous part means NO MATTER WHAT IS THE LAST MAJOR VALUE AT THE END OF THE URL (except for variable/value pairs and query strings), PUT THE '.format' AT THE END OF IT. Examples will show it clearly:

http://subdomain.domain.tld/module-or-resource/action.FORMAT

http://subdomain.domain.tld/fake_directory_name_for_user_readability/module-or-resource/action.FORMAT

http://subdomain.domain.tld/module-or-resource/action.FORMAT/var1#value1/var2#value2/

http://subdomain.domain.tld/fake_directory_name_for_user_readability/module-or-resource/action.FORMAT?not-user-friendly-query-string-value=something&also-serach-engines-dont-catalog-this-full-url=but-we-want-that

http://www.site.tld/resource/action/id.format

http://www.site.tld/products/ship/4d912f22c182293a70e2e7ac3671228dff397a52.FORMAT/carrier#ups/rate#blue/address#123-Mocking-Bird-Lane-Uphigh-CO-80230-USA/currency#usd


The '.format' is between the end of the part of the URL that selects the resource or code and the part of the URL that supplies variables to the code processing the request. (PHP/Symfony, PHP/Drupal, Ruby/Rails, et al. all use this)

This convention/practice was decided upon in my projects after going through the research below. I hope it is as useful to you as it was to me:

=========================================

=================
I looked at the following sites:
Twitter
yahoo
amazon S3
facebook
myspace
linkedin
google.

(EOL = End of the Last listed of Collection, Module, Action ,Or Id)

The methods of specifying the return type of the document were one of three different kinds:

1/ Implied, because only one was available (Amazon S3:json, Linkedin:xml)
2/ Implied but another could be specified BY ADDING ‘.format’ TO THE END OF THE EOL before any query string or friendly URL components at the end of the whole URL (MySpace-xml:add.json)
3/ Implied but another could be specified by a query string paramter (google-atom:‘alt=json’, yahoo-xml:‘output=json’)
4/ Format was required BY ADDING ‘.format’ TO THE END OF THE EOL ((Twitter:add .json|.xml|.rss)

=========================================

API Versions numbers used in URL of the the API for:
Yahoo
MySpace

EXAMMPLES of the way '.format' is added to API URLs:
-------------------------------------------------------
http://microformats.org/wiki/rest/urls
HTML
 GET /people/1
     return the first record in HTML format
 GET /people/1.html
     return the first record in HTML format
XML
 GET /people/1.xml
     return the first record in XML format
JSON
 GET /people/1.json
     return the first record in JSON format

http://confluence.sakaiproject.org/display/SAKDEV/EntityBroker+RESTful+URL+support

Access to an entity:(MEMBER)

   * http://localhost:8080/direct/webapp-entity/id0
   * http://localhost:8080/direct/webapp-entity/id0.xml
   * http://localhost:8080/direct/webapp-entity/id0.json

Access to an entity space:(COLLECTION)

   * http://localhost:8080/direct/webapp-entity
   * http://localhost:8080/direct/webapp-entity.xml
   * http://localhost:8080/direct/webapp-entity.json

Describing entities: (SELF-DISCOVERY/ERD)
   o http://localhost:8080/direct/describe
   o http://localhost:8080/direct/eval-evaluation/describe
   o http://localhost:8080/direct/webapp-entity/describe
   o http://localhost:8080/direct/webapp-entity/describe.xml

MySpace
EXAMPLE REQUEST:
   * XML: http://api.myspace.com/v1/users/454304609/albums
   * JSON: http://api.myspace.com/v1/users/454304609/albums.json

Twitter
http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search
Search(GET)
 o Example: http://search.twitter.com/search.json?callback=foo&q=twitter
            http://search.twitter.com/search.atom?lang=en&q=devo
Statuses/User_timeline

http://twitter.com/statuses/user_timeline/12345.xml or http://twitter.com/statuses/user_timeline/bob.json.
http://twitter.com/statuses/user_timeline.xml?user_id=1401881
http://twitter.com/statuses/user_timeline.xml?screen_name=101010
http://twitter.com/statuses/user_timeline.rss?page=3
Status Updates (POST)
curl -u user:password -d "status=playing with cURL and the Twitter API" http://twitter.com/statuses/update.xml


 
Further References
http://www.xml.com/pub/a/2004/12/01/restful-web.html
http://ajaxpatterns.org/RESTful_Service (EXCELLENT)*  Describing entities:
   o http://localhost:8080/direct/describe
   o http://localhost:8080/direct/eval-evaluation/describe
   o http://localhost:8080/direct/webapp-entity/describe
   o http://localhost:8080/direct/webapp-entity/describe.xml

Symfony recognizes four main http methods

Well, it turns out that Symfony gets loaded in all the methods except HEAD, as expected.

I tried:
-----------
GET
POST
DELETE
PUT
HEAD
OPTIONS

Symfony reported the method correctly in the ones that really matter, the first 4 above. In Options, it displayed GET as the method. In HEAD, it was never invoked, which agrees with the HTTP specification, so that's a good thing.

Well, that DOESN'T change the fact that PHP won't parse the body for DELETE or PUT requests and so there is no pretty access to the body. I'd have to come up with my own MIME/POST processor. Not going there :-(

So, I will use the POST for all PUTS, and probably most multiple DELETES. As I had originally talked about in a previous blog.

However, in Symfony, I need to deal with the crsf variable in the body of the request, and NO direct GET variables in the URL. Mr. Ponticier (creator of Symfony) has added an optional,standard POST/body-of-the-request variable for 'method'. I may or may not use that. Investigating now.

PS., one other thing that I would have to deal with is Symfony's labeling of variables, they already use the brackets for their use of an array of values for columns of a table.

And I tested once, PHP doesn't accept multiple sets of brackets for arrays of arrays in the names of POST variables.
Investigating.

One step back

Well, I had the code for doing a REST-POST(single or arrayed) as I have previously described completely done. Then I learned what SVN and backups should be good for :-(

I found out a way to switch two files, do my normal neurotic 1 save/min, and end up with only one of two files that I had written. Since I had been working on it 4-6 hours a day, in addition to my full time job, I was exhausted and despondent after doing it.

So, I stepped back, relaxed, and decided to skip the temporary functional style design and go straight to Symfony. I am reading all I can on REST a la Symfony. I don't totally agree with what I've seen, and especially, from the founder of Symfony. However, he is much better at using his framework than I am ;-) So I will keep an open mind as I try to do it, 'my way'. (see previous posts on this blog)

I am just about to see start a micro project to test two things:
1/ How to use the crsf token that Symfony generates in a REST environment.
2/ If the sfWebRequest object contains the Method used to access the script, and see if I can actually at least use single JSON DELETE/POST.

Someday, I am going to pay someone to hack PHP and allow the $_POST variables to be filled during POST/DELETE/PUT/GET methods.