Archive: February 2010

Sustainability has become an important topic for most major companies. Thousands of businesses around the globe currently track and publicly report their carbon footprint.  Initially, pressure from customers, employees and shareholders was a major driver for companies to monitor and report their energy and environmental performance. However, many companies now see sustainability projects as offering the highest ROI of any initiatives they undertake.

IT operations in large organizations typically consume a great deal of energy. Electricity can account for up to 15 to 20% of the data center operating costs. The latest statistics show that data center electricity consumption will double every 5 years. Data centers account for 2% of electricity consumption countrywide and this is growing. More than 30% of servers in the U.S. operate at less than 2% capacity and this is only one of the inefficiencies contributing to wasted electricity.

SolutionSet helps our clients meet the challenges and opportunities of optimizing energy consumption in IT operations:

Assessment and Definition:  we review your IT operation processes and current sustainability activities and recommend strategies and programs to bridge gaps and build upon opportunities.

Planning and Design: working with your internal teams we develop a plan focusing on the areas of greatest impact and ROI that may include:

• Underutilized server and other infrastructure resources
• Suboptimal hardware and applications requiring more hardware than necessary
• Overcooling and hotspots
• Expensive and non-renewable energy sources

Program Development and Implementation: based on these plans, we help you:

• Minimize underutilized server resources through virtualization (cloud computing) technologies
• Optimize servers and application to require less servers and consume less electricity
• Use alternative energy technologies such as photovoltaics, heat pumps, and evaporative cooling
• Use thermal profiling to identify hot spots and overcooling in data centers
• Implement energy usage reporting systems and applications–knowing how much you use is half the battle
• Develop applications to monitor energy usage not only in data centers but across the organization

Monitoring, Reporting and Refinement: We create dashboards that enable you to coordinate IT information from all over your organization, all over the world, then we flow the information into reports consistent with the reporting standards of the countries you serve.

Stakeholder Engagement: Where appropriate, we can develop online communities that enable you to share best practices with other IT operations professionals within your organization or with other companies. When approached correctly, IT sustainability projects can be extremely positive to your bottom line while contributing to your corporate responsibility objectives. Your relationships with employees, customers, partners and investors are enhanced through your commitment to sustainability.

By Amanda North, Pavel Pragin

A tour de force lasting from 9am to 7:30pm, day two of Symfony Live was packed with informative sessions and, of course, the preview release of Symfony 2.0 (which will be covered in its own separate blog post very soon).  I wrote this post based on my conference notes while on the plane back from Paris on precious little sleep, so please let me know if you find any inaccuracies. For day one of Symfony Live 2010, click here.

All of the presentation slides can be viewed online on the Symfony Live Event page on Joind.in.

Okapi and Symfony

The makers of the Okapi translation framework obtained an early copy of the Dependency Injection Container (DIC) from Sensio and undertook a migration of their product to use Symfony Components. The presentation gave some clues on the architecture of Symfony 2.0. For example, arguments will be passed explicitly to the controller (instead of the controller grabbing the request object itself). Lukas Kahwe Smith warned not to simply pass around the DIC as this defeats the purpose and nullifies performance gains. We were also given a peek into how the Dependency Injection (DI) configuration will use parameter syntax like %dsn% to avoid repetition. Symfony Events, as opppsed to the filter chain in symfony 1, can now call filters only once if necessary. They claim that migrating to the Symfony Routing component took only 2 hours to complete (plus some tweaks). After the migration, Okapi now relies more on Symfony classes than custom classes which means less code to maintain.

Optimizing PHP Code

Xavier de Cock’s presentation on performance was the most low-level in nature (from a technology stack perspective). The beginning of the talk was very Zend focused, and a bit hard to follow for a more application-focused person who is not very familiar with Zend Engine (also because of a thick French accent). Essentially, De Cock is interested in profiling an application, such as SwiftMailer, and analyzing every aspect down to the opcode in order to improve performance. In general, he discourages trying to “outsmart” the underlying opcode caching mechanism, noting that you may end up actually reducing performance. He uses Vulcan Logic Dumper to see what opcode was produced by the PHP code and employed both Xdebug and Zend Debug profilers to identify sections of code that are ripe for optimization.

Suggested tips and tricks

• Use built-in PHP functions (as opposed to your own custom functions)
• Use opcode caching (APC, eAccelerator, etc)
• Use data caching
• Optimize SQL (usually the number one culprit)
• Get your data from the 2DB in batches
• Create a PHP extension (I doubt most of us will I’ll go this far)
• Use other extensions like HipHop, Phc, Quercus or Roadsend PHP
• Pre-incrementation of a counter variable inside a loop performs better
• than post-incementation
• While loops outperform for loops
• Stay away from array functions like array_unique() (but they’re dang convenient)

Biggest common mistake: Creating memory leaks created by referencing objects in loops. This is only big deal in processing scripts like daemons (not typical web pages), and the cyclic garbage collection in PHP 5.3 provides vast improvement here.

Bottom line: the most interesting aspect of this talk was seeing what tools and techniques he used to identify slow code, but most developers should not worry about the majority of the techniques he demonstrated and just focus on optimizing database queries because that is where the low hanging fruit is.

Git 101

Scott Chacon is clearly a seasoned speaker, having done the Rails conference circuit, and the high quality of his presentation showed it. The graphics and animation in his slides are more than eye-candy; they genuinely help to demonstrate how Git works. Scott also gave a full-day Git training the day after Symfony Live, which I unfortunately could not attend. I reckon that this presentation was an abridged version of that training sans exercises. There were a lot of points that he did not have time to go into, but here are the high-level features:

• Developed by Linus Torvalds
• Used for Linux Kernel and Android
• Fully Distributed. Each clone is a backup and is equivalent
• No network connectivity needed to do work
• Immutable (never removes data)
• Based on full snapshots (not deltas)
• Commits are hashed strings (incrementing numbers, although convenient, are not used)

Parts

There are three primary working parts to wrap your head around:

1. Working Directory
2. Index
3. Repository

Workflow

In the Git workflow, there are four primary steps:

1. Edit Files
2. Stage your changes (git add)
3. Review your changes (git status)
4. Commit your changes (git commit)

Tips

If you’ve been working on a lot of different files and features simultaneously, committing a “changeset” of related edits instead of one gigantic commit is a good idea.

Random Thought by Scott

Although it is theoretically possible for two Git revision hashes to collide, it is more likely that your entire development team will be killed by wolves in separate incidents.

This presentation solidified my opinion that Git is generally better than Subversion and that it should be the SCM of choice in the future. I have been wanting to make the transition to Git as the default SCM for all of our PHP projects at SolutionSet, but existing infrastructure investments and lack of developer knowledge have been a hindrance. I hope that the use of Git for Symfony 2 will speed the adoption of Git in the PHP community and lead to SolutionSet’s transition to Git in 2010.

Writing Clean Class Interfaces with Symfony Events

Dennis Benkert, also the organizer of Symfony Day in Germany, shared his thoughts on how to reduce coupling in your classes and separate concerns. Accoring to Ted Faison, “Coupling is single greatest problem in complex system”. When different classes need to interact, a Service Layer should be used. There are two main ways to achieve decoupling (once you’ve already logically defined your classes): Depency Injection (DI) and Events. DI should be used for mandatory depencies and Events for optional dependencies. This presentation focused on Events.

Types of Event messages in Symfony

• notify - doesn’t expect a response
• filter - returns a value to the next filter in the chain
• notifyUntil - similar to a filter but stops the filter chain when a certain condition is met

When Dennis put out a call on twitter for example of Event usage, he got some useful responses such as allowing someone to override the behavior of a plugin, but the most notable was a sarcastic tweet about using Events to totally obfuscate your classes so no one else can follow them.

Event Usage Pitfalls

• Don’t use for mandatory coupling (use DI)
• There is no order to event listeners (if you are expecting this you are doing something wrong)
• Events make code harder to follow and debug
• Events can be slow once you add a lot of listeners
• Don’t use events in the model

There were a couple of memorable quotes from this talk…

A phrase that was picked up by multiple other speakers: “Every time you use sfContext, you kill a kitten”.

And my personal favorite (on the Symfony Events component mascot): “The octopus is keeping his eye on everything. Like with his arms.”

Zend Framework and Symfony

Matthew Weier O’Phinney, project lead for Zend Framework (ZF), started by saying “I am not the enemy”, and I believe him. Given the fact that Zend invited representatives of Symfony and other PHP frameworks (Cake, Agavi and CodeIgniter) to spar at the last ZendCon and the fact that Symfony 2 heavily relies on Zend Components, it appears that ZF and Symfony headed more in the direction of cooperation than competition. That being said, O’phinney still won the best tweet of the conference for saying “I enjoy the fact that Fabien is worried about spilling secrets to me”. Expect ZF to take a few pages out of Symfony 2’s playbook in the future.

Matt explained that there is really no magic to using Zend Components within Symfony; it’s as simple as adding some code for the Zend autoloader in the ProjectConfiguration class. An example of this is given in the Jobeet tutorial, but the code has been optimized to store the autoloader instance. His theory is use the best tools to do what you need to do regardless of their source, be it PEAR, EZ Components, or anywhere else.

Zend Library Standouts

• Feed Tools
• Remote APIs
• Lucene Search
• PDF generation
• Queuing
• Cloud computing

It was exciting to hear Matt talking about the Service Layer concept from Domain Driven Design. With the new flexibility of Doctrine 2 and Symfony 2, it should now be possible to design a richer domain layer within a Symfony project that includes Services, Entities, Data Mappers and a Data Access Layer (DAL). The Service layer is useful for validation and filtering, permissions (ACL) and interactions between Entities. For unit testing he mentioned 80% coverage as a good target (as opposed to 90-100%). Zend libraries can be used to easily expose your service layer as an API. I am looking forward to using this technique on a current project of mine.

Debugging Symfony

Alvaro Videla shared his experience debugging and optimizing a very large German dating site with 2 million members hosted on 28 production servers. He uses the vast amount of data that is stored in Symfony logs to identify slow areas of code by using a flag in APC to turn logging on in the production environment for a short period of time on each server and store the logs in CouchDB. CouchDB was able to handle storing 15 million log records in the first week!

Debugging Tools and Technologies Used

• AWK - for extracting log data
• avRedisLoggerPlugin - a persistent key-value database
• Tsung - a high performance, open source benchmarking framework
• XHProf - code profiler
• Graphite - a powerful data visualization tool

He uses logging to detect site outages by generating an alert if say 100 DB connection error logs happen within a minute and calls this “threshold logging”. Alvaro is also the developer of the FireSymfony Firefox plugin, which is an awesome replacement for Web Debug Toolbar except for the fact that it doesn’t work with the latest version of Firefox. Well, at least it doesn’t work for me on Snow Leopard and a lot of other people if you check the support forum.

Implementing a CMS in Symfony

When Marcos Labad set about providing a Content Management System (CMS) for his client in the publising business, he looked at the major options available in early 2009 (Sympal, Diem and Apostrophe) and decided to roll his own. Key aspects to the success of his project were great people, good communication, designating stakeholders and planning backwards.

Why Choose Symfony as base?

Active development and support Some Learning curve but good documentation Big and growing community Form framework Easy maintenance and support for the final owner Based on good practices and patterns Reusable parts Easy to integrate new engineers

Why not Drupal or Joomla? Not good for specific data models Learning curve as well May be fast to implement but are hard to maintain and extend

Tips

• Don’t forget about 301 redirects when upgrading a site
• Use Doctrine behaviors to save time

Symfony in the Cloud

Kris Wallsmith, release manager for symfony 1.3 and 1.4, is currently working on a startup called nebul.us which allows users to passively share what they are doing online (as opposed to actively blogging or tweeting about it). It seems like a powerful idea if they can make it easy enough to control what people see without making it an active process again. He made some very good points about deploying nebul.us (or your app) to the cloud:

1. You don’t need to know how the could works, you just need to know how to use it, and hosting providers like Rightscale, Amazon and others make it easy
2. There is nothing difficult about deploying to the cloud. It’s essentially the same LAMP stack you are used to.

Query Splitting

Kris put a lot of thought and into query splitting and has shared his work in the sfDoctrineMasterSlavePlugin which uses database transactions by default.

File Uploads in the Cloud

Files in the cloud will need to be:

• Uploadable to a service (like S3)
• Retrieved from a service in the view (with a helper)
• Disable-able (for the dev environment)

Deployment Tips

• Use the symlink method so you site is only down during the database migration (and not the code update)
• When using migrations during deployment, be sure to only execute once

Symfony at Yahoo!

Dustin Whittle joined Yahoo! with the assignment of migrating Yahoo! Bookmarks to symfony. Once Bookmarks was proven as a test case, Yahoo! went about migrating several more web properties, such as Yahoo! Answers, to symfony. Yahoo! Answers is the largest collection of human knowledge on the Internet (515 million answers). Clearly, a generic ORM is not going to cut it for a data store of this size, so Yahoo! only uses symfony for the presentation layer and uses web services to retrieve the data (although for smaller projects Yahoo! still uses Doctrine). Dustin makes a good point when he says, “all PHP developers use a framework, even if they don’t think so”. In other words, your non-framework based project still makes design choices that could be considered a framework. Dustin works for Rasmus Groth (the creator of PHP), who believes that you should design your application to solve your particular problem and nothing more. It was this kind of thinking that led Sensio to create Symfony 2 with much more flexibility.

Yahoo! Symfony Plugins

Yahoo! has created and shared a handful of plugins that they use to build symfony applications. They all start with a “y”, for example:

• ysfBuildPlugin (for deploying performant apps)
• ysfR3Plugin (for Yahoo!’s flavor of translation)
• ysfDimensionsPllugin (adding depth to symfony’s configuration)
• ysfYUIPlugin (for Yahoo!’s javascript library)

Tips and Observations

• Scalability is when you can add hardware and get a proportionate increase in performance
• Most performance comes not from the language, but from the design -Rasmus
• Don’t spend more time managing the cache than retrieving data from it
• Don’t use .htaccess (use regular Apache config for better performance)
• Aggregate and minify your CSS

Dustin encouraged developers to get to know the Yahoo! Open Stack.

MOST INTERESTING MOMENT: After plugging YUI, Dustin asked the room to raise hands to show what javascript framework they use. jQuery left all other libraries in the dust. Then he tweeted “It just clicked how many jQuery users there really are…”

Symfony 2

The preview release of Symfony 2 deserves it own post, which I will post soon.

For day one of Symfony Live 2010, click here.

Today was the first day of Symfony Live, the biggest symfony conference of the year and there has been a LOT going on. For day two of Symfony Live 2010, click here.

All of the presentation slides can be viewed online on the Symfony Live Event page on Joind.in.

sfPot

I don’t know if this event was named after Fabien Potencier (the leader of the symfony project) or what, but it was a good chance to drink some locally brewed beer with other serious members of the symfony community before the actual conference at Le Frog’s pub in a charming part of Paris.  I learned that Fabian thinks “Symfony 2 isn’t really an MVC framework in the typical sense”, but I’ll just wait until Wednesday’s unveiling of Symfony 2 to get the juicy details. Note: This event coupled with jet lag induced me to miss the first presentation the following day on internationalization.

Working with the Admin Generator

John Cleveley gave a useful and witty presentation on working with the symfony admin generator which included tips that are not in the documentation.  I did not know that the admin generator automatically created REST routes for your modules or the best techniques to go about creating your own admin theme.  Here are his “Ten Commandments”:

1. Understand the client’s workflow
2. Think about security from the start
3. Look through and understand the auto-generated, cached PHP files
4. Change the table_method call to reduce database calls
5. Use a custom (bespoke in U.K. english) form class for the admin if its different from your app(s)
6. Keep form configuration in the form classes (as opposed to the generator.yml) where possible
7. Create a theme or plugin to reuse your work
8. Consider users with small screens
9. Create functional tests to guard against regression
10. Maintain good MVC and decoupling practices

He also recommended the sfAdminDashPlugin and sfAdminThemeJsRollerPlugin.

Microsoft Presentation

Two representatives from Microsoft gave a pitch on the history of supporting PHP within Microsoft, the Open Source Techonology Center (OSTC) and using Windows Azure for cloud applications.

Symfony Internals

Geoffrey Bachelet quickly walked the audience through symfony’s execution of a request in a french accent so thick you could cut it with guillotine.  This was a review for advanced developers but a very useful reminder of the relevant design patterns that symfony implements such as the Observer and Chain of Responsibility.  It was enlightening to learn that returning the “Error” template is not often used since the new form framework was introduced and that the proper way to serve ajax requests for HTML is with the “None” return value in your action.

Doctrine Migrations

Dennis Benkert, the orgnizer of Symfony Day in Germany, shared his knowledge of Doctrine migrations which are largely inspired by Rails migrations.  I was pleased to learn that Doctrine now has command line tasks that will compare your new schema.yml file to your existing model classes and generate a “migration file” for you.  This migration file is associated with a numbered and timestamped version of the database and consists of an up() and down() method which you run on your production database at deploy time.  This will replace the risky, error-prone and annoying process of having each developer maintain a SQL file with schema changes.  It is worth noting now that rollbacks don’t often work if a failure occurs during the migration and that Doctrine 2 will use a totally new migration technique.

Doctrine 2

The most exciting presentation of the day was given by Jon Wage on Doctrine 2.  With all of the talk recently about the Propel project being reinvigorated of late, this was Jon’s chance to show that Doctrine is still in the leading spot as far as PHP ORMs (Object Relational Mappers) go.  Given the features that Jon showed and that Fabien and Sensio are currently supporting Doctrine, I would agree that Doctrine is definitely the ORM to choose when starting a new symfony project or choose which one to learn.

The codebase has been completely rewritten to take advantage of PHP 5.3 and performs a LOT faster according to the benchmarks reported. Fabien requested that they “Remove the magic”, so there will be a bit more work for developers to do, but a better understanding of what is really happening and more control for advanced queries. A key feature is the separation of the ORM and the DBAL (Database Abstraction Layer). This will allow for the DBAL to used as a standalone component if desired and will allow for schema-to-database comparisons and improved migrations.  DQL (Doctrine Query Language) will be a true language with a recursive parser that builds queries and throws useful, informative exceptions.  A more explicit relationship between your model classes (based on PHP comments) will vastly improve entity inheritance, performance and the ability to write raw SQL and still get hydrated objects back as a result.

Worst case timeline for a stable release is six to twelve months.

Offline Admin Generator with HTML5 and Gears

Thomas Parisot’s offline admin generator is a peek into the future of running web applications offline.  Neither HTML5 nor Google Gears (which is discontinued) is supported enough to make this a reality for a consumer application yet…perhaps if you can force people to use a particular browser.  He did make some interesting observations about how it is possible make a javascript listener on every form that posts to the server to generically capture and locally store all of the transactions in the local SQLite database.  There are two models to choose from when designing an offline app.  The simpler approach is to have the user explicitly request to switch to “offline” mode before they lose connectivity.  The preferable, yet more complex approach is to always store transactions locally and then check if the connection is still up when moving from the local copy to the database.  Admittedly, the sample offline admin generator was more of a proof-of-concept than anything we can expect to actually use in the near future.

The Symfony Community

Finally, Stefan Koopmanschap, the current symfony community manager, gave a talk about how to properly get involved in the symfony community.  There are the Google Groups (symfony-users, symfony-docs, symfony-devs and symfony-community).  If you can’t find what you want there, then there are the IRC channels or sending out a tweet with the #symfony hashtag.  There are many local groups as well.  For example, the San Francisco Symfony Meetup group where I’ll be giving a recap of Symfony Live in the near future.  He stressed that if no one contributes, then there is nothing to be gained from the community, so please submit bugs to Trac if you find them, answer people’s questions, speak at a meetup or conference and introduce yourselves to your fellow developers!

For day two of Symfony Live 2010, click here.

We’re firm believers in the power of utilizing behavioral data to solve business problems. With the availability of reliable quantitative data, goals like marketing optimization and efficiency become a reality. Under the old database marketing paradigm, life was straightforward: Every marketing dollar was tracked against resulting customer-specific, transactional metrics. Inquiries, registrations, purchases, and profit could all be tied to addressable prospects and customers, and the cost of soliciting those actions could be calculated with precision. What we learned about consumer behavior informed our efforts, and the outcome was, in most cases, measurable improvement in our marketing efficiency over time.

But recently, the avalanche of new online data has threatened to derail us, cluttering our minds and our marketing dashboards with an ever-expanding array of disparate tables and graphs. A significant amount of time and effort is invested in compiling and reporting on these new metrics. But are all of them really advancing us toward our goal? Impressions, clicks, page views—where did the customers and sales go?

Author and business consultant Eric Ries wrote a post for the Harvard Business Review titled Entrepreneurs Beware of Vanity Metrics that really gets to the heart of the matter.  In it, he presents a discussion about Actionable metrics and their nemesis: Vanity metrics.  Whether you are a Start-up or Big Company, we think his observations apply to the multichannel marketing arena.  We agree with him that metrics should be:

Actionable—If you don’t believe it or you can’t repeat it, you can’t be confident in your decisions.
Accessible—To support effective business decision-making, reports must be understandable, available, and timely.
Auditable—Metrics must be credible, with no loss of integrity between the individual customers and atomic activities that roll up into summary numbers that appear in reports.

The Web brings us a lot of instant gratification and a mountain of data. As Mr. Ries accurately points out, people have a tendency to believe (and take credit for) positive metrics while they excuse (and deflect responsibility for) negative ones.  The most successful organizations will be those with the objectivity to discriminate between the actionable and the feel-good data. In other words, let’s be picky about what we measure and separate metrics that capture the essence of consumer behavior from metrics that reflect our own marketing department’s output.  If we maintain our focus and believe in our metrics—positive and negative—our business decision-making will be cheaper, clearer, and better.

Last week the Securities and Exchange Commission ruled for the first time that public companies must alert investors of any serious risks that their businesses may face due to global warming. Environmental issues have been among the threats that the SEC has required companies to disclose, but this is the first time that global warming specifically has been called out. This is just the latest indicator that “green” is becoming a standard and necessary consideration in business practice. Business leaders across industries proactively are analyzing the impact of their activities, developing strategies aimed at improving efficiency, engaging their employees in the process and reporting the results to their various constituencies.

Digital technology is a key enabler for businesses in their quest for sustainability. Digital dashboards aggregate enterprise data and compare it to standards set by leaders in their industry and, increasingly, by regulatory authorities. Business information systems analyze the information and focus decision makers on optimizing consumption of resources like water, paper and power. Online communities encourage employees to participate in making their areas more efficient and share tips. Social networking media technologies enable companies to share their results with broad constituencies and invite discussion. New C level positions are being created to oversee this process. In addition to the societal benefits they are generating through investment in sustainability projects, these executives report very positive impacts to their bottom line.

Digital technologies and services are powerful tools for companies exploring sustainability and engaging in these practices. In the area of paper consumption alone digital communications can reduce or eliminate paper by helping companies migrate to online investor and corporate responsibility reports, using electronic forms, creating online communities to support customers, partners and developers, and shifting to “eCirculars” and web-based information instead of printed materials distributed by the US Mail.

Businesses who pursue digitally based sustainability strategies succeed in achieving the dual goals of: “adopting business strategies and activities that meet the needs of the enterprise and its stakeholders today while protecting, sustaining and enhancing the human and natural resources that will be needed in the future.” (World Commission on Environment and Development)

See the New York Times editorial 01.31.10 about the SEC ruling.

• Ages 8-18 online more than 7.5 hours each day
• Map of Netflix rental patterns across America
• Most only see 1% of the 140,000 iPhone apps
• Write next chapter in Rosetta Stone farm boy ad