I followed Cloudera’s instructions for setting up CDH3 scripts on the Amazon EC2 instances I was testing.

Everything went great. Until I got to the Whirr installation (which seems the easiest way to start up a number of nodes at once and have them auto-magically configured.)

Following these instructions gave me this error: “Non-Windows AMIs with a virtualization type of ‘hvm’ currently may only be used with Cluster Compute instance types.”

Luckily a search for this error message lead to these helpful links:

• http://mail-archives.apache.org/mod_mbox/incubator-whirr-user/201103.mbox/%3CD890DBF7-9434-47AD-BA1E-E492099473A6@daltonclark.com%3E
• http://mail-archives.apache.org/mod_mbox/whirr-user/201112.mbox/%3CCAHZL8y8OncJ3tkWYfgwTnPP_6ziW8D02Z0s8u5-XST+1rv79yw@mail.gmail.com%3E

After some trial and error, I added these lines to my hadoop.properties which worked:

==========
whirr.hardware-id=m1.large
whirr.image-id=us-east-1/ami-da0cf8b3
whirr.location-id=us-east-1
==========

Once there, everything worked great. One small thing is that when the whirr script completes setting up the instances, it will say that the web-based interfaces are live, but you will have to edit the security groups in AWS to accept incoming traffic from any IP address:
50030 0.0.0.0/0
50070 0.0.0.0/0
in order to actually see the web-based interface.

So without much ado, here is the PHP script to set up an instance and get its hostname, as well as output these as messages using the AWS PHP SDK.

	// Sleep time to allow EC2 instance to start up
	$sleeptime = 15;
	$username = "ec2-user";

	// For AWS PHP SDK
	putenv('HOME=/home/ec2-user/');
	require_once 'AWSSDKforPHP/sdk.class.php';

	// Get data from HTTP POST
	$ami = $_POST['amis'];
	$instancetype = $_POST['instancetype'];
	$keyname = $_POST['key'];
	$securitygroup = $_POST['securitygroups'];

	// Instantiate the AmazonEC2 class
	$ec2 = new AmazonEC2();

	// Boot an instance of the image
	$response = $ec2->run_instances($ami, 1, 1, array(
	    'KeyName' => $keyname,
	    'InstanceType' => $instancetype,
	    'SecurityGroupId' => $securitygroup,
	));
	if (!($response->isOK())) {
		echo "<p class='error'>ERROR! Could not create new instance!</p>";
		return;	
	}
	$instance = $response->body->instancesSet->item->instanceId;
	$message = "<p>Your instance has been successfully created.</p>";
	$message .= ("<p>Instance ID is: <b>$instance</b></p>");

	// Give instance some time to start up
	sleep ($sleeptime);

	// Get the hostname from a call to the DescribeImages operation.
	$response = $ec2->describe_instances(array(
		'Filter' => array(
        array('Name' => 'instance-id', 'Value' => "$instance"),
		)
	));
	if (!($response->isOK())) {
		echo "<p class='error'>ERROR! Could not retrieve hostname for instance!</p>";
		return;	
	}
	$hostname = $response->body->reservationSet->item->instancesSet->item->dnsName;

	// Output the message
	$message .= "<p>Your instance hostname is: <b>$hostname</b></p>";
	$message .= "<p>You can connect to your instance using this command:<br>" .
		"<b>ssh -i $keyname.pem $username@" . $hostname .  "</b></p>";

	echo $message;

I wanted to have a UI system on the backend that allowed me to create and edit users, and set/reset their passwords in a manner similar to how its done with most professional sites these days:

• Enter the password twice for creating a user
• Enter password twice for editing a user, but blank password means password is unchanged

With this in mind, I made a few changes to the users models, controllers and views.

1. MODEL VALIDATION

First of all, I changed the validation code for the users.php model to the following:

	var $validate = array(
// other validation code
		'password1' => array(
			'notempty' => array(
				'rule' => array('notempty'),
			),
		),
		'password2' => array(
			'notempty' => array(
				'rule' => array('notempty'),
			),
		),
	);

Basically removing all hints of the “password” field. Why? As you will see later, I’ve removed any sort of “view” or direct access to the password in the views and controller.

2. VIEWS CHANGE FOR ADD/EDIT

Next for the views, I made these changes to the add and edit views.

I replaced

		echo $this->Form->input('password');

add.ctp

		echo $this->Form->input('password1', 
			array('label' => 'Password',
				'type' => 'password')
		);
		echo $this->Form->input('password2', 
			array('label' => 'Password (again)',
				'type' => 'password')
		);

edit.ctp

		echo $this->Form->input('password3', 
			array('label' => 'Password',
				'type' => 'password')
		);
		echo $this->Form->input('password4', 
			array('label' => 'Password (again)',
				'type' => 'password')
		);

Why password1 and password2 for add.ctp, and password3 and password4 for edit.ctp ? Because as you might remember from the model, password1 and password2 are validated – which means anyone using the UI MUST enter the password for the field. Changing the fieldnames makes it easy for the edit view – so people can leave those fields blank.

3. CONTROLLER CHECKS FOR PASSWORD

Finally, the controller. Here is how to handle the add() method. Between these lines:

	function add() {
		if (!empty($this->data)) {
			// add following code here
			$this->User->create();

add these lines:

			if (!($this->data['User']['password1'] === 
				$this->data['User']['password2'])) {
				$this->Session->setFlash(__('Passwords do not match.', true));				
				return;
			}
			$this->data['User']['password'] = 
				$this->Auth->password($this->data['User']['password1']);

For edit(), its a little trickier since you want to check for blank passwords, and if so, leave the user password unchanged.

	function edit($id = null) {
		... ...
		if (!empty($this->data)) {
			// add following code here
			if ($this->User->save($this->data)) {

add these lines:

			// Passwords do not match
			if (!($this->data['User']['password3'] === 
				$this->data['User']['password4'])) {
				$this->Session->setFlash(__('Passwords do not match.', true));				
				return;
			}
			// If the password is left blank, 
			// just reuse the existing password
			if (strlen($this->data['User']['password3']) == 0) {
				$User = $this->User->read(null, $id);
				$this->data['User']['password'] = $User['User']['password'];
			} else {
				$this->data['User']['password'] = 
					$this->Auth->password($this->data['User']['password3']);
			}

And that’s it!!! A professional UI for adding users, combined with CakePHP’s great in-built authentication system!

UPDATE: Oct 1, 1930 EST: In order to view the username and provide a logoff option add this to app_controller.php to provide the variable to all your views:

	function beforeRender() { 
		$this->set('trUsername', $this->Auth->user("username")); 
	} 

and this line to somewhere in your views:

	<p class="login">Logged in as <b><? echo $trUsername ?></b>
	[<?php echo $this->Html->link(__('logout', true), array('controller' => 'users', 'action' => 'logout')); ?>]
	</p>

Dokuwiki stores its user info in conf/users.auth.php file. You can generate a list of users from a plain text or other file in this format:


john.doe
jane.doe
john1.doe
jane2.doe


and then run the following PHP script (say, name it, dokuusers.php) and output the result to a text file, and move the files as below:

$ php dokuusers.php > users.txt
(copy-paste the user list here)
$ cat users.txt users.auth.php > users2.auth.php
$ mv users2.auth.php users.auth.php


The dokuusers.php file is below.

<?
/* Customize these lines */
// Company e-mail domain
$companyDomain = "company.com";
// Group(s) to add users to
$groups = "user";
// Function that returns password string
// First example: john.doe has password john11111
function passwd ($string) {
        return $string . "11111";
}

$f = fopen('php://stdin', 'r');
while ($line = fgets($f)) {
        $line = trim($line);
        list($first, $last) = explode(".", $line);
        $passwd = md5(passwd($first));
        $name = ucfirst ($first) . " " . ucfirst ($last);
        $email = $line . "@" . $companyDomain;

        echo "$line:$passwd:$name:$email:$groups\n";
}
?>

Installation XML

You can keep the name (plugin.xml).
Make sure to change the first two lines and the last line:

<?xml version="1.0" encoding="iso-8859-1"?>
<mosinstall version="1.0" type="mambots" group="content">
...
</mosinstall>

to:

<?xml version="1.0" encoding="utf-8"?>
<install version="1.5" type="plugin" group="content">
...
</install>

Add an author e-mail as well:

<authorEmail>aaa@aaa.com</authorEmail>

Change the attribute from “mambot” to “plugin”:

<filename mambot="etplugin">etplugin.php</filename>

<filename plugin="etplugin">etplugin.php</filename>

Plugin code

Keep the same file name : plugin.php

Change the first line from:

defined( '_VALID_MOS' ) or die( 'Direct Access to this location is not allowed.' );

to

defined( '_JEXEC' ) or die( 'Direct Access to this location is not allowed.' );

Change registerFunction() to registerEvent():

$_MAMBOTS->registerFunction( 'onPrepareContent', 'botFnCall' );

$mainframe->registerEvent( 'onPrepareContent', 'plgFnCall' );

Make sure to change references for variables such as $mosConfig_live_site, $mosConfig_absolute_path, $database, etc as specified here.

The function call construct no longer has four arguments:

function botFnCall( $published, &$row, $mask=0, $page=0  ) {

Instead it has three:

function plgFnCall( &$row, &$params, $page=0 ) {

Add a performance check for good measure:

	// simple performance check to determine whether bot should process further
	if ( JString::strpos( $row->text, 'pluginTag' ) === false ) {
		return true;
	}

Change the published check:

	if (!$published) {
		return true;
	}

to:

	if ( !$params->get( 'enabled', 1 ) ) {
		return true;
	}

Change any references to the “/mambots/” directory to “/plugins/” (if you’re including custom CSS, JS, and the like.)

Lazy loading of images is great. And there’s even a jQuery plugin for lazy loading.

The one problem is: I don’t want my images to load only during a scroll – I want lazy loading, but want the images loaded right after the Javascript kicks in.

So I did some experimenting, and was able to replace:

$("img.lazyload").lazyload();

with this:

$("img.lazyload").each(function() {
    $(this).attr("src", $(this).attr("original"));
    $(this).removeAttr("original");
});

And that’s it! Lazy loading of images on your website, but without requiring the scroll. (You can delete the removeAttr() line if you want, I just like my rendered HTML to be clean.)

And best of all, you don’t need a jQuery plugin to achieve this – it can be done with just the basic jQuery library.

Note that the above requires the patched version of lazy-loading, which only works when your original HTML code for images looks like this:

<img src="maybe_1x1_image_placeholder.png" original="original_image.jpg" alt="" />

Yes, it breaks HTML semantics, but is probably the only way to achieve lazy loading on modern browsers. More info below.

More on Lazy Loading

Lazy loading of images is an awesome idea. The idea behind lazy loading of images is to load images for a website after the page is fully loading, thus enabling the browser to fully load the page even without all images completely loaded. And lazy loading loads the images when required.

Problem is, it doesn’t work on modern browsers. This is a pity, since I think that plugin is really cool, and is well written.

There are some patches that work, but this requires one to swap the original “src” attribute of the image with another fake one (called “original”) and then let the Javascript work its magic during load time.

While this breaks webpage semantics, its apparently the only way to get things to work on current browsers, particularly Webkit-based ones (Safari, Chrome). (And it may break SEO functionality, since you are replacing img src and pointing to a fake image, and the replacement/load is done in Javascript.)

But that didn’t stop me, since it works well and loads my site faster. :)

This is the PHP script that I wrote recently to automate some of my downloads. Note that there are two pre-requisites to be able to run this script:

• PHP command-line program (php5-cli) has to be installed
• file_get_contents() has to allow remote URL fetches

Without much ado, here is the script.

<?
/*
 * THIS SCRIPT DOWNLOADS A SEQUENCE OF FILES FROM THE WEB
 * TO THE LOCAL DRIVE.
 * Released under GPL.
 * Author: Suman Srinivasan, Dec 24, 2010
 */

/* Replace the parameters below with the names you need.
 * The "{x}" part of the string pattern will be replaced by the 
 * number in the script. */

/* The initial parameters */
// Source URL pattern
$sourceURLOriginal = "http://www.somewebsite.com/document{x}.pdf";
// Destination folder
$destinationFolder = ".";
// Destination file name pattern
$destinationFileNameOriginal = "doc{x}.pdf";
// Start number
$start = 1;
// End number
$end = 10;

// From start to end
for ($i=$start; $i<=$end; $i++) {
	// Replace source URL parameter with number
	$sourceURL = str_replace("{x}", $i, $sourceURLOriginal);
	// Destination file name
	$destinationFile = $destinationFolder . "/" . 
		str_replace("{x}", $i, $destinationFileNameOriginal);
	// Read from URL, write to file
	file_put_contents($destinationFile, 
		file_get_contents($sourceURL)
	);
	// Output progress
	echo "File #$i complete\n";
}
?>

I was able to find the code to handle this in actions.php in the root of my Dokuwiki install. In that file, there is a function called function act_auth($act). In that function, look for this piece of code:

  //handle logout
  if($act=='logout'){

That’s the one that handles the action when users log out.

The last line of that control sequence is:

    act_redirect($ID,'login');

You just need to replace $ID with “”, and that redirects all users who are logging off to the front/home page. So when they log back in, they are on the start page.

New code:

    act_redirect("",'login');

Three VMWare nodes running on one machine. This is the network topology to configure.

I had to set up the following:

• Content server: public IP address with Apache installed
• NetServ node: a router with one public IP and one private IP
• Client: Private IP, connected through the NetServ node gateway

I need the client to access the outside world only through the NetServ router.

First, I downloaded a Fedora 10 VMWare image from thoughtpolice.co.uk and configured it as I needed. Then I made three copies of it.

[Note about clean-start: When copying the VMWare images to create clones, make sure to shutdown the images first before copying. The reason is that when you copy and start the clone, VMWare changes the MAC address and other settings, and you need to clean-start the VM to get the new MAC address and other settings to apply.]

I then created a few virtual network configurations to get my network topology right. A screenshot of the Virtual Network Editor is below.

The Virtual Network Editor showing the three VMNets used in the experiment.

• I have three physical network adapters. I configured each of them to map to one particular VMNet. VMNet0 is bridged to the first physical adapter (that is my host machine).
  • The two other VMNets – which are bridged to the physical adapters – are the ones used by the public network adapters for the Content Server and the NetServ node.
• For the private network, I created host-only VMNet with DHCP service, which basically provides a private network.
  • The private network adapter for the NetServ node and the Client are mapped to this VMNet.

Settings for Client VMWare network adapter.

So far, so good. Boot up the nodes and they should get correct IP addresses and:

• Content Server and Netserv Node can connect to the public Internet.
• Client can only ping the Netserv node’s private IP address.

Now to get the full network topology working.

For this, we need to allow NetServ node to run as a router and a DNS server. Here is how to do so (all as root):

• /etc/init.d/dnsmasq start

• echo 1 > /proc/sys/net/ipv4/ip_forward

• Several iptables rules

Now, on the client machine, you need to set up the default gateway to be the NetServ router:

• route add default gw NETSERV_IP_ADDR eth0

[UPDATE Sep 24 2010] On the client machine, you also need to:

• sudo vi /etc/resolv.conf

and add the NetServ/gateway machine as the default DNS server.

And that’s it : you should have a complete network topology up and running! Try a few pings and opening a few websites, and it should all work!

Instead of using a MySQL database, I simply used an associative array and serialize() and unserialize().

So my code basically looked something like this:

if (numberOfUrl ($url) == 0)
  do_something_for_first_request();
else
  do_something_for_subsequent_requests();

But the function numberOfUrl($url) needs persistent storage – and MySQL would have been overkill.

So I used a simple associative array and serialize() and unserialize() to a file.

/**
* Returns the number of times this URL has been requested
*/
function numberOfUrl ($url) {
  // The file name
  $arrayFile = "urls.db";

  // Read the file, unserialize it to an array
  $urlArray = unserialize (file_get_contents($arrayFile));

  // If this URL exists in the array, just increment it
  if (isset ($urlArray[$url]))
    $urlArray[$url]++;
  // Otherwise add a new entry and set it to 1
  else
    $urlArray[$url] = 1;

  // Write out the serialized array back to disk
  file_put_contents ($arrayFile, serialize($urlArray));

  // Return the number of accesses
  return $urlArray[$url] - 1;
}

Of course, you will have to implement some form of mutual exclusion or locks to ensure that the whole thing happens as a transaction, especially if this PHP code can be accessed by multiple clients at the same time.