How to Order Categories by Most Recently Updated in WordPress

To display categories, it has wp_list_categories function, where You can order the categories by ID, name, slug, count or term_group. I want the categories ordered by most recently updated. Most recently updated I mean, category that has the latest post will display first in the list.

PHP

<?php
	$cat_array = array();
	$args=array(
	  'post_type' => 'post',
	  'post_status' => 'publish',
	  'posts_per_page' => 20,
	  'caller_get_posts'=> 1
	  );
	$my_query = null;
	$my_query = new WP_Query($args);
	if( $my_query->have_posts() ) {
	  while ($my_query->have_posts()) : $my_query->the_post();
		$cat_args=array('orderby' => 'none');
		$cats = wp_get_post_terms( $post->ID , 'category', $cat_args);
		foreach($cats as $cat) {
		  $cat_array[$cat->term_id] = $cat->term_id;
		}
	  endwhile;
	}
	if ($cat_array) {
	  foreach($cat_array as $cat) {
		$category = get_term_by('ID',$cat, 'category');
		echo '<a href="' . esc_attr(get_term_link($category, 'category')) . '" title="' . sprintf( __( "View all posts in %s" ), $category->name ) . '" ' . '>' . $category->name.'</a>'.'<br />';
	  }
	}
	wp_reset_query();
?>