Pages

Friday, December 13, 2013

Leadership Speeches

I usually do not read books or listen to the Babas when I need some inspiration or motivation. I believe one can not give speech if they have not experienced it. I don't want to read books like "15 ways to win" unless the author have achieved something. I always look up to the sportsmen whom I have followed right from my childhood. Recently I saw some of them and all of them were on leadership. This well applies to at our work also. Being a lead we lead a team of few and have to answer to the management at the same time. (whom its probably are harder to explain) In the same way the team captain needs to lead the players and at the same time need to answer the board and people who probably never played the game. I am embedding only the 1st parts of the video , do find the other parts in the related video section.

  1. Nasser Hussain : One of the most attacking  players of my time. He led England cricket team from the front in many times and personally I find his British accent the most soothing.

  2. Wassim Akram : He lead a team who is not even safe in its own country if lost some matches.

  3. Finally Sourav Ganguly : I admired him the most for changing the attitude of indian cricket team when he came .

Friday, November 29, 2013

Some Popular Songs of Y2K

Came across some very popular songs during Y2K. Decided to make a public playlist with the songs those were very frequently requested to the MTV request show presented by Nikhil Chinappa then. Feel free to remind me some more.

Friday, November 22, 2013

My response to a job applicant

Lately I have been looking for some mid/senior level Java programmers for my team in Hyderabad studio. I got resumes from a lot of sources and decided to respond to some of them with feedback before forwarding them to the talent acquisition department. I took some time two write a lot of details in the feedback because I have been observing a trend in resume writing which do not seem to be correct to me. Here is one of the personal feedback I write to a applicant of my acquaintance  .

Thanks for sending the resume. Sorry for the delay in reply . I was too busy with office work. At this moment I am not able to forward your resume as it looked to me that the openings I have in my company did not match your profile. However I will keep your resume and  let you know if I find any junior level opening in future. Meanwhile you can consider following points to increase your chances of resume being shortlisted.

These are my general and personal opinion.





  1. Communication, when you email someone for career or office work be careful about following general English mistakes.

    1. Initial/Address should be in Caps e.g. "Hello" or "Hi" (not "hello" or "hi")

    2. Try to address by name e.g. "Hello Neil" . Tech companies do not encourage "Sir", it may sound funny/sarcastic sometimes.

    3. Name should start with upper case. e.g. "Neil" not "neil", "poonam" etc

    4. I would not know what "ASE" is, its your company designation . Either write in full or just write generic titles like "Developer" or "Engineer".

    5. Always check the grammer.

    6. Be careful about the company name you are applying for. I guess you wanted to write "EA" and wrote 'xx' instead. Try to do research about company from internet before you apply.

    7. Avoid short form/SMS language like "pls".

    8. Do not put bullet points when not required like 1, 2... etc in your email. The body started with bullets.



  2. Things to avoid in resume

    1. "potential of implementing java development application software programs" does not mean anything. It could be just "Java programming or development".

    2. What is "oracle coding standards" ? Its not a coding language . It could be PL/SQL. Do not write something which you are not sure of. You will be asked questions and if not answered , it will disappoint the interviewer.

    3. Again, "thorough approach to the work" does not mean anything.

    4. "Currently working" and "till date" in the same sentence does not mean anything.

    5. Nobody works on MS-DOS now and if you put specific ones like Windows8, that may mean you cant work with Windows7 and Linux. OS is not required to be mentioned at all in these kind of jobs.

    6. JavaScript is a single word, if you put space, reviewer will think you are faking it. Same with HTML (all upper case).

    7. In project summery , no one would be interested to know functional details like "MultiVu only officially supports SFU and MDU accounts with PRIN ". Just write technical challenges you solved without going too much company specific. Also remember no one like developers who leak proprietary information.

    8. A resume reviewer will definitely not have any idea about "xxxxx event name at XXXX college". If you want to write about it, you can write specific competitions you won.

    9. "Runners in cricket tournament" means you were just substitute runner for a batsmen. :)

    10. Statements like "Capable of adopting myself to different environment." is too generic. Never write them.

    11. Most important - Read the resume few times if that sounds good.




These are only my observations in the email and resume. There are many more small things to be careful about. However there would be some mistakes without your knowledge, those are fine as long as you know what is correct.


Again, these are my personal opinion, I may sound very picky but just wanted to give you heads up before you send the resume somewhere else. Please take these feedback positively and work on it. Employers like people who are ready to change for good.


Feel free to email me anytime.


All the best

Neil

Thursday, November 21, 2013

Monopoly Bingo

Electronic Arts recently launched a new mobile game called Monopoly Bingo in Android and iOS. Fortunately I got a chance to work on it. In my own opinion it has excellent graphics and game play experience is a lot of fun. The game can be downloaded from Google Play and Apple Appstore. Try it out and feel free to give feedback/rating . Also you can like the Facebook page and follow the twitter  (@MonopolyBingo)account to get updates about new stuff related to the game.



Disclaimer - Opinions expressed are solely my own and do not express the views or opinions of my employer.

Moved my blog to VPS

Few months back I had moved my blog to Heroku, now I wanted to try out VPS. Heroku was all good but updating every thing via Git commands was kind of a pain when there is a image upload or wordpress version upgrade. VPS is like renting virtual hardware and OS and getting full access to the same. I choose DigitalOcean as they were very cheap and some of my friends gave good feedback about it. I choose the lowest plan that was 5 USD per month for a 512 MB ram. I got root access to the virtual machine and installed LAMP (Linux, Apache, MySQL and PHP) to host my PHP blog.

Blog ran all fine initially but soon I started getting the errors as MySQL kept shutting down automatically. So i had to login and start it manually. From error log and customer support I came to know that for tech stack I am running 512Mb may not be sufficient, so I decided to give it some swap space. Anyway DigitalOcean claims all its storage is SSD and those are supposed to be faster than HDD. Following are the commands for setting up swap.

dd if=/dev/zero of=/swapfile bs=1M count=1024

mkswap /swapfile

swapon /swapfile

# vi /etc/fstab

(add line /swapfile swap swap defaults 0 0)

swapon -s

free -k

This reminded me late 90s linux installation when we had to allocate some space to swap partition in the hard disk apart from boot and root (most of the times in addition a partition containing Windows 98 :) ) . of course those days RAM used to be expensive.

The blog has been running fine after that, I will continue to monitor for few for days.

 

Monday, November 18, 2013

Cultural shocks ? Not really !

At one of the work parties, someone asked me "What were the cultural shocks did you get after you came to USA ?" Its really a valid question and while I was trying to think of an answer I concluded that there were so many things different but not all of them are really "shock".America

Most of them were not actually shock because we do follow a lot of American movies and sitcoms. But few things I really discovered after landing here which was not very common in India, neither they show it in movies. (May be because it is very trivial here.)

  • American roads gives priority to the pedestrians. So if you are in the side of the road the card generally do not cross until you cross the road. DO NOT TRY in India. No vehicle stops for pedestrian to cross.
  • Be it hotel or office, if you run into someone whom you don't even know and make eye contact , you gave to smile and at least say "hello" or "Good Morning". Sometimes even known people are ignored when in such situations happen in India. Moreover in India people can continue to stare at you without any business as if very interested  in your activities. Staring would be definitely rude in America.
  • Most of the places I saw the queue starts after a bit of space from the counter and no one tries to jump the queue. However jumping queue considered to be "smart" in india and people feel proud about it. I have seen many of my acquaintances  feeling proud and showing off that they managed to get something done before someone else who deserved the same. Ideally they should feel ashamed about it and I find it as a form of corruption.

Overall I found America as I expected.Not that everything they do is perfect (i.e. keeping shop light turned on all night and wasting a lot of food in restaurants ), there is definitely so many things we need to learn to incorporate in out system.

Saturday, August 17, 2013

Recursively Find Hyperlinks In A Website

I was trying to write a script to crawl a website and fetch all the hyper links pointing to all the a particular file type e.g. .pdf or .mp3. Somehow the following command did not work for me.
wget -r -A .pdf <URL>

It did not go recursively and download all PDF files. I may have to ask in  stackoverflow.

Anyway I wrote my script in python and it worked well. At least for the site I was trying crawl. The following scripts give all the absolute URLs pointing to the desired type of files in the whole website. You may have to add few more strings in excludeList configuration variable to suite your target site else you have end up infinite loop.

[code language="python"]
import re
import urllib2
import urllib

## Configurations
# The starting point
baseURL = <home page url>
maxLinks = 1000
excludeList = ["None","/","./","#top"]
fileType = ".pdf"
outFile = "links.txt"

#Gloab list of links already visited , don't want to get into loop
vlinks = []
#This is where output is stored the list of files
files = []

# A recursive function which takes a url and adds the outpit links in the global
# output list.

def findFiles( baseURL ):
#URL encoding
baseURL = urllib.quote(baseURL, safe="/:=&?#+!$,;'@()*[]")
print "Scanning URL "+baseURL

#Check maximum number of links you want to store
print "Number of link stored - " + str(len(files))
if(len(files) > maxLinks):
return

# the current page
website = ""
try:
website = urllib2.urlopen(baseURL)
except urllib2.HTTPError, e:
print baseURL + " NOT FOUND"
return
# HTML content of the current page
html = website.read()
# fetch the anchor tags using regular expression from the html
# Beautifull Soup does it wonderfully in one go
links = re.findall('(?<=href=["']).*?(?=["'])', html)
#
for link in links:
#print link
url = str(link)
# Found the file type, then store and move to the next link
if(url.endswith(fileType)):
print "file link stored" + url
files.append(url)
f = open(outFile, 'a')
f.write(url+"n")
f.close
continue
# Exlude external links and self links , else it will keep looping
if not (url.startswith("http") or ( url in excludeList ) ):
#Build the absolute URL and show it !
print "abs url = " + baseURL.partition('?')[0].rpartition('/')[0]+"/"+url
absURL = baseURL.partition('?')[0].rpartition('/')[0]+"/"+ url
#Do not revisit the URL
if not (absURL in vlinks):
vlinks.append(absURL)
findFiles(absURL)
return

#Finally call the function
findFiles(baseURL)
print files
[/code]

Sunday, August 11, 2013

Getting started with XBMC

XBMC is a free and open source software media player for various OS platforms especially mobile. This is very useful to convert your TV dongles e.g. Android PC or Apple TV or Raspberry PI to a media center in your TV. This can not only organize and play your local media but also can stream movies and TV series. I downloaded the Android APK and installed my RocketChip MK806 Android TV.



To begin with I scanned all my mp3 and videos in my SD card using XBMC and all of them were ready to play. I found couple of plugins which listed almost all recent movies and TV series very well organized by season and episodes. As of now I have installed

  1. Mash Up ( Installation Steps )

  2. 1 Channel ( Installation Steps )




Making it Full screen : However you can notice that the android navigation bar at the bottom always appears (Even during movie play). This is sometimes distracting. So I found an app Fullscreen which can help you get rid of this navigation bar.

Start on Boot : Also If XBMC is the only app you are going to use every time you boot your android TV then it makes sense to have it launched automatically every time you boot your device. I found this app Android Startup Manager. In fact you could disable all the user and system app which you think is not of your use in the android. For me the only app I want android to run is XMBC because all other app I use in my phone.

Remote Control : Having a remote control app in your mobile for XMBC is very important, because some of the feature does not work from a wireless mouse. The official remote app is good but I use the app Yaste because it has a feature of seek bar of current video being played. However I am unable to make it auto discover the IP of the XBMC device yet.

Tuesday, August 6, 2013

Moving the blog again

A year ago I moved from free shared Linux hosting to paid one to GoDaddy. Everything was good, there was no downtime like free hosting solutions. WHen I was in free hosting sometimes my site got blocked by antivirus software because somebody else would have hosted such content in the same server.  However I did not earn any revenue from this blog so paying for hosting was not really my favorite idea.

I read several articles in the internet why GoDaddy is not a very good choice for hosting blogs. I had also used GoDaddy for hosting the website of local chapter of IEEE section (ieeehyd.org) but faced a lot of problem during renewal. First of all only the 1st year hosting price was attractive but the renewal charge was almost four times than that I initially paid for starting the hosting. I had my credit card in GoDaddy payment methods and now they wouldn't let me remove it until I gave details of another card. This inspired me to close the account itself. Also lately my site got blocked by websense several times , probably because of their blacklisted servers.

I had considered to move to wordpress.com (the PaaS solution ) but they were charging a lot for assigning the domain name and they did not have a domain transfer facility so that I could delete all my account and GoDaddy and only pay wordpress for the domain name. I was also not happy with dealing with GoDaddy customer support who kept on replying me  with some templates from their manual book instead of looking into my hosting problem.

I came across an old post of one of my friend which shows how an wordpress blog could be installed in heroku even in the free tier. Ofcourse there were some limitations but I wanted to get rid of current hosting. Apart from the steps mentioned in the blog post I had to do some additional steps to overcome the limitations of heroku and have my blog up and running.

  1. Heroku now have ClearDB for MySQL so I did not have to go for heroku's posgreSQL service as mentioned in the blog post.

  2. Because mine is not a new blog and I was moving all the contents from Linux host. I used the wordpress export file and wordpress-importer plugin to migrate the database. For images stored in wp-content  I downloaded via ftp and pushed using git from my local directory. However I was able to reduce the size of the content by using some orphan image checker plugin and deleting unattached files.

  3. I had to create  the .htaccess at the root directory of my blog at heroku to make sure the permalink's of post and pages are redirected to the appropriate query strings. This is normally created by the wordpress installer itself but heroku file system can not be altered permanently unless attached to storage service which is paid.

  4. The DB size was small , so I added a plugin to optimize the wordpress  database once in a while. It also made the website fast. Initially I saw my export file is huge but later realized that there was a post in which I had dragged and dropped a lot of images from my desktop. All those were stored as data-uri scheme (plenty of junk characters in the post text itself) instead of separate image files.

  5. The MySQL user created by default by clearDB did not have insert and update grants in the database so the database upgrade did not go through after I manually upgraded the wordpress to 3.6.

  6. GoDaddy DNS manager did not have the option to forward the  the domain name (with masking )  to the heroku's url where my blog was hosted. It was accepting only the IP address which heroku did not give me. So I forwarded it to a subdomain (www.neilghosh.com) and created a CNAME record to forward the domain to the subdomain and subsequently to the heroku URL.


Following are the challenged I am ready to face during maintenance of the site but I think its worth it, if I could save some money. On the bright side you get used to git commands because of frequent usage :) .

  1. Every plugin/theme needs to added via the local git repository to the wp-contents/plugins directory because anything uploaded via the admin portal will not be persisted in storage. It may seem so temporarily but eventually they will go missing when heroku moves the app as part of load balancing. Paid Amazon S3 storage is recommended but as of now I am pushing everything via git. I will try to see of I can use Dropbox for this.

  2. Similarly pictures in the posts should be uploaded through git, otherwise it may be hosted in some 3rd party site and embedded in the blog. This is a better option because it saves the blog's bandwidth and if blog is moved , the HTML code still referees to the same image.


Thanks to my colleague Sridhar  who initially game the idea of heroku. I think I will be happy with it for sometime till my hunger to pay around it stops. If I could generate some money out of it , Google Compute Engine and AWS/VPS are definitely some good areas to play around. I also like the idea of static blogs so that I don't waste computing (DB operations and PHP interpretation ) every time some user requests a page from my blog.

Monday, July 22, 2013

Screen scraping using YQL

I have been using Yahoo pipes for a long time now. I have done some screen scraping mash up using them. While yahoo pipe provides a component to fetch the HTML content from a URL, it is bit difficult to cut a specific part because it totally relies on a string to match. I came across YQL console where I could write SQL like queries and fetch the HTML content of any URL. The best part was that it supports XPath expressions for selecting the exact node of the HTML to extract data. For example I write the following query to get the stock price from the web page

[code language="sql"]
select *
from html
where url ="http://getquote.icicidirect.com/NewSiteTrading/trading/equity/includes/trading_stock_quote.asp?Symbol=BSES"
and xpath='//td[p/text()="LAST TRADE PRICE"]/following-sibling::td[2]/p'
[/code]

See above code running in YQL Console.

Similarly this query can be made little bit complex and parametrized for the stock symbol to form the appropriate url

[code language="sql"]
select *
from html
where url in (
select url
from uritemplate
where template="http://getquote.icicidirect.com/NewSiteTrading/trading/equity/includes
/trading_stock_quote.asp?Symbol={item}" and item=@item)
and xpath='//td[p/text()="LAST TRADE PRICE"]/following-sibling::td[2]/p | //td[p/text()="LAST TRADED TIME"]/following-sibling::td[1]/p'
[/code]

See the above code in YQL console. However this will not directly run from the console. One could just create a query alias and pass the required query string like the following.

http://query.yahooapis.com/v1/public/yql/neilghosh/liveQuote?item=INFTEC

This could have been done using the built in YQL component in the Yahoo Pipes itself but it would be an extra layer if you just need to get the required content from the HTML instead of having to play around any feed (for which Pipes is still the best choice). Of course there some limits/quota while using such YQL queries, which I need to explore in coming days.

For screen scraping I could directly use Google App Engine's URLFetch or curl in PHP servers but this would unnecessarily transfer the whole content consuming quota and leading to time lag.

Monday, July 1, 2013

Sharing Photosphere

I have been always a fan of panorama images. There are a lot of photo stitching software which can join many overlapping images to create a single one. I have used Photosynth earlier with a lot of satisfaction. There a lot of phones and digital cameras which can do it right out of the camera in panoramic mode, in which you have to slowly move the capturing device and it will continuously take and stitch photos to create larger panorama.  While there ares some websites like CleVR and GigaPan can help sharing the horizontal panoramas and let embed in various sites, spherical panoramas like the ones taken from photosphere app in Android 4.2 cameras could not be embedded in a straight forward way without doing some HTML coding. Finally I found a site called SphereShare.net for this purpose. Following is the embedded photo I took uploaded in this site.

(Click this link for a wider view. Because of low width of the blog, embedded one may not look good)



This time the players were moving, I will try to get a more stable image when I go outdoors next time :)
Here is a horizontal panorama embedded in photosynth. (Needs Microsoft Silverlight plugin )

Play music remotely from phone using Windows 7

I was searching for a Bluetooth transmitter which can transmit the music played in my cellphone to the speaker which mounted on my wall, so that I could control what is being played from my phone without using a very long aux cable. I had earlier posted about this but today I found it really useful to play and control music while lying down on bed. After pairing the phone you just have to click on the following setting in window to transmit music from phone to the speaker via the laptop. I could tune to any internet radio station , play any music on my phone and also take calls in the phone and everybody in the house could hear it.

BT1

Saturday, June 29, 2013

Night Trek at Khajaguda (!Supermoon)

Yes it was supermoon night but cloud cover ruined it it. While it is proved that supermoon is one of the most over credited astronomical event , still its one of the reason to go out and enjoy the nature at night. Thanks to city light, tough there was no moonlight , we could see the path and do the trek without a torch. We climbed some rocks as well (scrambling). It was monsoon and great time to trek.Thanks to organizers of hats club for holding this meetup and Rajesh for the photos from his new Sony DSLR.

DSC07083

Wednesday, June 26, 2013

Google AppEngine Channel API

I am not a big fan of page refresh for getting new data from server. AJAX has been the go to technology for these kind of requirements. However its unidirectional. For two way communication where server push is required , we have seen many technologies like comet etc. Websocket and WebRTC has been really cool technologies which helps sending data from server and client real time. These needs special server code for handling such requests. I have already worked on JSR 356 for websocket and glassfish reference implementation (tyrus) earlier. However in Google Appengine provides channel APIs for bidirectional communication. While client to server communication is still over HTTP GET or POST, sever creates a specific "channel" and enables itself to push data any time to specific clients. Under the hood , its actually the client which keeps polling with GET requests for new data to be sent by the server. In any case this API can be use fill in real time game servers.

The server side code

I added the following code to the example given in the previous post to use channel APIs

[code language="java"]
private void boradCastNotes(Request request,Note note) {
ServletContext context = hsr.getSession().getServletContext();
HashMap<String,ChannelPresence> liveUsers = (HashMap<String,ChannelPresence>)context.getAttribute("liveUsers");
if(liveUsers != null){
ChannelService channelService = ChannelServiceFactory.getChannelService();
ObjectMapper mapper = new ObjectMapper();
System.out.println("List of connected client ... ");
String noteStr = null;
try {
noteStr = mapper.writeValueAsString(note);
} catch (IOException e) {
e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
}
for(ChannelPresence cp : liveUsers.values()){
System.out.println(cp.clientId());
System.out.print(" Sending message to client --> " +noteStr);
String channelMessageStr="{"command":"note","data":"+noteStr+"}";
System.out.print(" Sending message to client after wrapping --> " +channelMessageStr);
channelService.sendMessage(new ChannelMessage(cp.clientId(),channelMessageStr));

}
}
}
[/code]

Client Code

In HTML I added an extra button which will send the user entered data using an AJAX request and some JavaScript code to create the channel using the token issues by the server earlier.

[code language="JavaScript"]
<script language="JavaScript">

//Function called when update button us pressed
//This will maken an AJAX POST request
//To the webservice created using sitebricks
postnote = function(){
var noteObj = new Object();
noteObj.text=document.forms[0]['note.text'].value;
sendMessage('/notes','POST',JSON.stringify(noteObj));
};
//Generic method for sending any Ajax request
sendMessage = function(path, method,param) {
var xhr = new XMLHttpRequest();
xhr.open(method, path, true);
//Callback when response is received from the server
xhr.onload = function () {
console.log(this.responseText);
document.forms[0]['note.text'].value="";
};
xhr.send(param);
};

onOpened = function() {
sendMessage('/notes','GET','command=open');
};

onMessage = function(message){
//When message is recived from the server
//Message.data contains the actual string send by the
//Java code
//Now convert the JSON string to a Javascript Object
var data = eval("(" + message.data + ")");
console.log("Received data from Server "+data)
//Adds a new note row in the tables
if(data.command=="note"){
insertRow(data.data)
}
}

insertRow = function(data){
var table=document.getElementById("noteTable");
var row=table.insertRow(1);
var cell1=row.insertCell(0);
var cell2=row.insertCell(1);
cell1.innerHTML=new Date(data.date);
cell2.innerHTML=data.text;
}

//Opens a channel with server with the given token (provided by the server)
//Internally it keeps polling the server for new messages
channel = new goog.appengine.Channel('${token}');
socket = channel.open();
//Define all the listeners
socket.onopen = onOpened;
socket.onmessage = onMessage;
socket.onerror = onError;
socket.onclose = onClose;
</script>
[/code]

Other features

To track the clients which gets connected to the server , listeners can be added to a specific urls.I track the clients to broadcast the messages to all the connected clients.To enable tracking following needs to be added to appengine-web.xml

[code language="XML"]
<inbound-services>
<service>channel_presence</service>
</inbound-services>
[/code]

Then write POST endpoint handlers

[code language="Java"]
public class TrackerServlet extends HttpServlet {
@Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
ChannelService channelService = ChannelServiceFactory.getChannelService();
ChannelPresence presence = channelService.parsePresence(req);
System.out.print("Client trying to connect with ID " + presence.clientId());
//Save the new client in servlet context
ServletContext context = getServletContext();
//Object liveUsers = context.getAttribute("liveUsers");
HashMap<String, ChannelPresence> liveUsers = (HashMap<String, ChannelPresence>) context.getAttribute("liveUsers");
if (null == liveUsers) {
System.out.println("Initialising client list");
liveUsers = new HashMap<String, ChannelPresence>();
context.setAttribute("liveUsers", liveUsers);
}
if(liveUsers.containsKey(presence.clientId())) {
System.out.println("Err.... this guy was already connected ! ");
} else {
liveUsers.put(presence.clientId(), presence);
System.out.println(" New client connected with ID " + presence.clientId());
}
}
[/code]


Similarly remove the client from servlet context when client gets disconnected.

[code language="Java"]
public class TrackerServlet1 extends HttpServlet {
@Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
ChannelService channelService = ChannelServiceFactory.getChannelService();
ChannelPresence presence = channelService.parsePresence(req);
System.out.print("Client disconnected with ID " + presence.clientId());

ServletContext context = getServletContext();
HashMap<String, ChannelPresence> liveUsers = (HashMap<String, ChannelPresence>) context.getAttribute("liveUsers");
if (null != liveUsers) {
if (liveUsers.containsKey(presence.clientId())) {
liveUsers.remove(presence.clientId());
System.out.println("Client was disconnected");
} else {
System.out.println("Client was not connected");
}
} else {
System.out.println("No client was ever connected");
}
}
}
[/code]


Note:

 

Thursday, June 20, 2013

Appengine.Maven.Guice.Sitebricks.Objectify

I have been trying to read and use all these technology for my project but I did not find any single article/blog which includes setup instructions for all of them.

Most of these technologies are from Google and optimizes for their PaaS solution.

  • Google AppEngine - The platform as a service supporting several languages including  Python, Java.

  • Maven  - Build tool like ant but lets you pull the libraries from the original repository dynamically at the build time.

  • Guice - Dependency Injection tool like spring without XML configuration. Configuration is done in code itself.

  • Sitebricks -  Libraries for creating REST webservices and dynamic HTML page (separating HTML and Data ).

  • Objectify - Library to interact with Google AppEngine datastore and automatic memcached management.


Following are the major steps I took for creating a working project.

  1. Since I was trying to create a web app with maven I needed to create the folder structure for web project containing WEB-INF etc. I could do this manually too. Details .
    $ mvn archetype:generate -DgroupId=com.neil -DartifactId=NoteWebApp -DarchetypeArtifactId=maven-archetype-webapp -DinteractiveMode=false


  2. Added the Google appengine dependency in the POM.xml in the recently created Maven project.
    [code language="xml"]
    <dependency>
    <groupId>com.google.appengine</groupId>
    <artifactId>appengine-api-1.0-sdk</artifactId>
    <version>1.8.1</version>
    </dependency>
    [/code]

  3. Add Google app engine plugin for the maven tools to be used for running devserver and uploading the build to appengine cloud.
    Here we could also mention the debug port which any idea can connect.

    • Local server can be started using "mvn appengine:devserver". Details.

    • Local app can be deployed in appengine server at "mvn appengine:update"


    Sometimes "port in use" error occurs if previous debug port or server port is not closed gracefully then we have to query for the process using the port and kill it.
    sudo lsof -i :8080 # checks port 8080 in mac
    kill -9 2828

    [code language="xml"]
    <plugins>
    <plugin>
    <groupId>com.google.appengine</groupId>
    <artifactId>appengine-maven-plugin</artifactId>
    <version>1.8.1</version>
    <configuration>
    <!--
    <jvmFlags>
    <jvmFlag>-Xdebug</jvmFlag>
    <jvmFlag>-agentlib:jdwp=transport=dt_socket,address=5000,server=y,suspend=n</jvmFlag>
    </jvmFlags>
    -->
    </configuration>
    </plugin>
    </plugins>

    [/code]

  4. Add the appengine-web.xml in the same directory as web.xml which will hold the appengine configuration.
    [code language="xml"]
    <appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <!-- create this unique id in appengine console in web-->
    <application>testjavaneil</application>
    <!-- I only keep only one version during dev and keep overwriting it. -->
    <version>1</version>
    <!-- I have no idea about it -->
    <threadsafe>true</threadsafe>
    </appengine-web-app>
    [/code]

  5. Add dependency for Guice
    [code language="xml"]
    <dependency>
    <groupId>com.google.inject</groupId>
    <artifactId>guice</artifactId>
    <version>3.0</version>
    </dependency>
    [/code]

  6. Add a listener servlet in web.xml which will get executed when container comes up. Also add a filter which will redirect all the urls to the Guice filter
    which in turn will have the information which class to be executed depending on the request URL.This configuration will be done in the listener.
    [code language="xml"]
    <filter>
    <filter-name>webFilter</filter-name>
    <filter-class>com.google.inject.servlet.GuiceFilter</filter-class>
    </filter>

    <filter-mapping>
    <filter-name>webFilter</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>

    <listener>
    <listener-class>com.neil.MyGuiceServletConfig</listener-class>
    </listener>
    [/code]

  7. As promised in the last step I will create the listener with the information of url mappings
    [code language="java"]
    public class MyGuiceServletConfig extends GuiceServletContextListener {

    @Override
    protected Injector getInjector() {
    return Guice.createInjector(
    //Keep sending Guice the modules
    new SitebricksModule() {
    @Override
    protected void configureSitebricks() {
    scan(NotebookService.class.getPackage());
    //Should change this to logger, this is just to proove that
    //sitebrick scans the classes for annotations like @At etc.
    System.out.println("****** Scan complete ******");
    }
    }
    , new ServletModule() {
    @Override
    protected void configureServlets() {
    //Servlet classes have to be singleton to be consistent with servlet specification
    //In tranditional cases web.xml config tell the container to do so I guess.
    bind(com.neil.NotebookServlet.class).in(Singleton.class);
    //Analogous to typcial servlet URL mappings
    serve("/servlet").with(com.neil.NotebookServlet.class);
    }
    }
    );
    }
    }

    [/code]

  8. Following is a typical servlet class but will will avoid this and use templates and webservices to render data in HTML or JSON format.
    [code language="java"]

    public class NotebookServlet extends HttpServlet {

    //Register the entity class for data persistance service
    static {
    ObjectifyService.register(Note.class);
    }

    /**
    * Get requests come here
    *
    * @param req
    * @param resp
    * @throws ServletException
    * @throws IOException
    */
    @Override
    public void doGet(
    HttpServletRequest req, HttpServletResponse resp)
    throws ServletException, IOException {
    resp.setContentType("text/html");
    //Render the form for adding notes
    resp.getWriter().println(
    "<form method=post action="/servlet" >" +
    "<input name="note.text" size="20" type=text/>" +
    "<input type=submit value="Add Note">" +
    "</form>");

    resp.getWriter().println("List of notes");

    //load all the data from datastore
    List<Note> notes = ObjectifyService.ofy().load().type(Note.class).list();
    //Render the notes in each row of the table.
    resp.getWriter().println("<table><tr style="background:grey"><th>Date</th><th>Note</th></tr>");
    for (Note noteEntry : notes) {
    resp.getWriter().println("<tr>");
    resp.getWriter().println("<td>" + noteEntry.getDate().toString() + "</td><td>" + noteEntry.getText() + "</td>");
    resp.getWriter().println("</tr>");
    }
    resp.getWriter().println("<table>");
    }

    /**
    * Handles the form submit post requests and redirect to the same get request to display the list of
    * notes
    * @param req
    * @param resp
    * @throws ServletException
    * @throws IOException
    */
    @Override
    public void doPost(HttpServletRequest req, HttpServletResponse resp)
    throws ServletException, IOException {
    Note note = new Note();
    note.setDate(new Date());
    note.setText(req.getParameter("note.text"));
    ObjectifyService.ofy().save().entities(note).now();
    doGet(req, resp);

    }

    }
    [/code]

  9. Create the entity to map the database
    [code language="java"]
    @Entity
    public class Note {
    @Id
    private Long id;
    private Date date;

    public Long getId() {
    return id;
    }

    public void setId(Long id) {
    this.id = id;
    }

    private String text;

    //rest of the getters and setters
    }
    [/code]

  10. Create the class with sitebrick and annotate as service. annotate the url mapping and get post methods
    [code language="java"]
    @At("/notes")
    @Service
    public class NotebookService {

    private Note note = new Note();
    //Register the entity class for the Objectify persistance service.
    public NotebookService() {
    ObjectifyService.register(Note.class);
    }

    @Get
    Reply<List<Note>> showNotes() {
    //Prepare the HTTP headers
    Map<String, String> headers = new HashMap<String, String>();
    headers.put("Content-Type", "application/json");
    //Fetch data from database
    List<Note> notes = ObjectifyService.ofy().load().type(Note.class).list();
    //Convert the entity object to JSON and return.
    return Reply.with(notes).as(Json.class).headers(headers);
    }

    public Note getNote() {
    return note;
    }

    public void setNote(Note note) {
    this.note = note;
    }

    /**
    * Post request endpoint here inserts data in database
    * and returns the JSON of the single entry which was newly
    * created
    *
    * @param request the body of the request containing data is
    * obtained from here.
    * @return
    */
    @Post
    public Reply postNote(Request request) {
    Map<String, String> headers = new HashMap<String, String>();
    headers.put("Content-Type", "application/json");
    //Read JSON data and create entity object
    note = request.read(Note.class).as(Json.class);
    //System generated date instead of user
    note.setDate(new Date());
    //Store data
    ObjectifyService.ofy().save().entities(note).now();
    //Just return the newly added data
    return Reply.with(note).as(Json.class).headers(headers);
    /*
    //to redirect to a url but this class is only for
    //webservice call, we don't ahve to redirect to any page
    return Reply.saying().redirect("/servlet");
    */
    }
    }
    [/code]

  11. For using sitebricks with HTML template create another class and HTML
    [code language="java"]
    @At("/webnotes")
    @Show("/notes.html")
    public class Webnote {

    //When HTML page is rendered , this instance variable would be used for the placeholders to populate
    private List<Note> notes;
    //Following instance variable will be populated when form is submitted from the HTML template
    //The name of the input fields will be mapped to the entity components
    private Note note = new Note();

    @Get
    public void get() {
    this.notes = ObjectifyService.ofy().load().type(Note.class).list(); //load from db
    }

    public Note getNote() {
    return note;
    }

    public void setNote(Note note) {
    this.note = note;
    }

    @Post
    public String post() {
    //Date is not provided by the form, server date is populayted
    note.setDate(new Date());
    ObjectifyService.ofy().save().entities(note).now();
    //Redirect to same class and render the same HTML template
    return "webnotes";
    }

    public List<Note> getNotes() {
    return notes;
    }

    public void setNotes(List<Note> notes) {
    this.notes = notes;
    }
    }

    [/code]
    [code language="java"]
    <!--
    HTML Template for sitebricks. This has the html form and table to enter and display the data
    However theer are placeholders for sitebricks to replace the data before serving to client
    -->
    <html>
    <head>
    <title></title>
    </head>
    <body>
    <form method=post action="/webnotes">
    <input name=note.text size=20 type=text/>
    <input type=submit value="Add Note">
    </form>
    <br>

    <table>
    <tr style="background:grey">
    <th>Date</th>
    <th>Note</th>
    </tr>
    @Repeat(items=notes, var="note")
    <tr>
    <td>${note.date}</td>
    <td>${note.text}</td>
    </tr>
    </table>
    </body>
    </html>
    [/code]

  12. Add the objectify dependencies in POM.xml
    [code language="xml"]
    <dependency>
    <groupId>com.googlecode.objectify</groupId>
    <artifactId>objectify</artifactId>
    <version>4.0b3</version>
    </dependency>
    [/code]

  13. Register the objectify Entity class as service whenever we create any service which will have database interaction. I do it in constructor of the service class
    [code language="java"]
    public NotebookService() {
    ObjectifyService.register(Note.class);
    }
    </li>
    [/code]

  14. Read/write data using Objectify. Note.class is the entity class
    [code language="java"]
    List<Note> notes = ObjectifyService.ofy().load().type(Note.class).list();
    [/code]


    [code language="java"]
    ObjectifyService.ofy().save().entities(note).now();
    [/code]


I have put all the code in Github and its still evolving, however you can get hold of the particular commit.

Sunday, April 28, 2013

Giving a new life to old smartphone

My Samsung Galaxy Ace's battery has become very week lately. It has swollen and been discharging really fast. Whenever I used to connect charger , the battery charge level used go up to 100% within minutes and again discharge within an hour when I start using it. It got discharged within minutes from 100% to 0% while using WI-FI or 3G.

Initially I thought , it may be due to age of the phone (2 years), rooting the phone, installing custom ROM (manufacturer knows the phone best ) or due to partitioning SD card to increase internal storage (phone may be unknowingly accessing my slow class 4 SD card frequently). I also consulted some "Gadget Guru" friends of mine but as expected their knowledge was limited to the user's and market point of view. They asked me to get a new phone as a 3 year old would have suggested me :). I completely forgot that fact that in our childhood gadgets we used to change the rechargeable batteries. These days the smart phones are being upgraded so frequently nobody bothers to get a new battery.

Finally I decided to replace the battery for 450 INR and then get a new phone if the phone really had a problem. It has been a week and battery performance has been drastically increased  as if it is a new phone. Here is a screenshot of the battery usage after I charged the battery to only 85%. I read somewhere the battery life increases more when you let the battery discharge below 10% and then charge all the way till 100% (one has to wait even after 100% , phone should say battery is charged.)
Also note that following good practices helps the phone retain charge for longer hours in my experience


  • Keep the WI-FI turned off when not in wi-fi area, phone spends a lot of power just searching for signal

  • Disable Background data use when not required.

  • Create profiles and manage various power settings depending on the usage (Home, work or Car). This feature is available in most of the custom ROMs.

  • Reduce screen brightness  at night time.

  • Close the apps with Back button of Android because they keep running in the background if you press home button. Apple products does magic in this case by putting the background apps to sleep.


Sunday, March 24, 2013

Increase your Internal Phone Storage

One of my biggest problem with my phone got solved today. I had been using an old Android phone with only 181 Mb of Internal phone storage. As a result my phone starts complaining after installing few apps, especially the ones which can not be moved to SD card. I have a 16GB SD card installed but its of no use when it comes to running more number of apps. Lately I had to uninstall a lot of my favorite apps and rely on the browser, e.g. I never could install Android apps for Facebook, Twitter, Google+ etc. I always used the mobile websites.

Finally I started searching the workaround for the same. I found that a part of my SD card can be partitioned and formatted as linux file system , so that Android OS considers it as part of phone storage. This is possible for phones which are rooted and I was already using custom ROM from Cyanogenmod.

Galaxy ACE

Note: following will wipe all your data in SD card including music and photos.

  1. Use Mini Tool Partition Wizard Manager to delete the original SD card partition and create two primary partitions as follows

    1. A FAT32 partition with most of the size of the SD card.

    2. An EXT2 (old type file system format to make sure most of the ROM support it), you can try with EXT3 or EXT4 also but I never tested. I kept this partition as 1 GB, I think I will never run out of this space , adding apps.



  2. Click "apply" to commit the partition changes  and format the SD card.

  3. Copy and unzip the system.zip file on to the SD card.

  4. Put the SD card back in the phone and start the Terminal Emulator app of the rooted phone.

  5. copy the file 40int2ext file (found in the system.zip file ) in /system/mnt/etc/init.d directory of the phone. I used the "cp" command and changed the permission of the destination directory for writing file after changing the user to super user (su). You may have to click "allow" while changing the user.

  6. That's it, I rebooted and phone and saw the internal memory has been increased to almost 1 GB.


Wow ! that was great, I immediately loaded the phone with all the apps I always wanted to install.
I was surprised to see that the performance did not decrease noticeably even tough it is a low class SD card. I will continue to test by installing more apps.

Finally a word of caution : Do this only if you fine in case the phone or the card does not work after this experiment.
However if experiment is successful , you will never see the "insufficient storage" or "low memory" error :)

Source

Friday, March 8, 2013

Recursively find list of files in a directory in Python 2.2

I have been using python for writing small utility programs. Till now I avoided scripting languages like python because I though shell script is master of everything and works in any linux computer. They also have all programming constructs like loops and conditions etc. However when it came to quickly get some work done, shell script too a lot of time write a error free program. Finally I jumped to python to do some file batch processing.

The most fundamental need to batch file processing is to iterate through all the files in a given directory. I could found several ways to do so using os.walk function but my old linux server had only Python 2.2 and I did not want to FTP files to my local PC to server back and fourth . So here is the code which also works in old version of python.

 
[code language="python"]
import os

# finds all the files in a directory recursively
def listAll(dirName):
filelist1=[]
files = os.listdir(dirName)
for f in files:
if os.path.isfile(os.path.join(dirName,f)):
filelist1.append(os.path.join(dirName,f))
else:
newlist=listAll(os.path.join(dirName,f));
filelist1.extend(newlist)
return filelist1
# root directory path
rootdir= '/home/'

for f in listAll(rootdir):
print "Absolute Path " + os.path.abspath(f)
print "File Name" + os.path.basename(f)
print "Directory Path" + os.path.dirname(f)
# Change directory
os.chdir(os.path.dirname(f))
# print current directory
os.system("pwd")
print os.getcwd()
# check file extension
if not ( f.endswith(".java") or or f.endswith(".xml")):
# Do nothing
continue


[/code]

Wednesday, February 27, 2013

Born to code ? Or its just me ?



I don't remember how many of my friends and colleagues who are qualified engineers and doing the top technology jobs told me that they are bored of their "IT Job" and probably want to join a bank or may be do a cloth business. I always wanted to ask them why didn't you think about this when you joined Engineering school ? or at  least deciding the major in engineering school ? Like computer science or mechanical engineering ? When I was deciding the same back in 2002 people almost forced me to take Mechanical or Electrical Engineering just because there was a dot com boom in the technology industry. Even at that time I believed that new possibilities are in computer science are way more than what there is in all other fields of engineering combined. By the time I graduated things really changed and I observed that graduated of all other departments of engineering taking up technology jobs and starting to write code for the first time in their lives. Thanks to the companies who only give you puzzles instead of programming question. Had I known that , I could have just practiced puzzles instead of learning to code all 4 years of my undergrad !

I love the above video. It says there aren't many qualified programmer to fulfill upcoming requirements. Then why is there so much of unemployment despite of thousands of engineering colleges around all offering Computer Science and Information Technology related degree ? I don't even see any private engineering college investing in heavy machinery for Electrical or Mechanical engineering departments. I found my answer when I toured various engineering colleges for guest lectures. Actually its just the degree. Even a final year student does not raise hand when I ask "Have you written any program in any language ?" I doubt its because they are sky or lack in communication skills. Anyway I highly recommend this video.

Anyway I always felt that I am born to code and this is what I think about few others :)


However all above are just my opinion. Not trying to belittle any branch of Engineering. I still love electronics, machines and electrical circuits, they are the basis of all computers and its real life applications. Just that I am happy that I studied what I wanted to do and doing what I wanted to study. I am just fortunate to realize this early , most people realize late and hard work in the past remains useless.

Friday, February 15, 2013

Astroid 2012 DA14 tonight !

As you may have come across news articles that asteroid 2012 DA14 will be fly by earth very closely. It will in fact come closer than the Moon and some Geo stationary satellites.

You can see the live event in SLOOH SpaceCamera if you think you are in a wrong timezone.

Following is the heaven's above chart.



Ajay Talwar did a fantastic job by calculating the  speed of the asteroid relative to the sky which is very helpful to spot it using a telescope. Note that you need at least a small telescope or good binocular to spot it even when it is closest because the apparent magnitude would be little beyond what unaided human can see at the darkest location.

Here is the approximate schedule created by Ajay Talwar considering the viewer from New Delhi

15th February
23:53 IST - 2012DA14 rises in the constellation of Centaurus, apparent speed 0.44 degrees per minute, Mag 8.8
16th February
00:27 IST - crosses into Hydra, apparent speed 0.71 degrees per minute, Mag 0.78
00:29 IST - close to beta Hya, apparent speed 0.72 degrees per minute, Mag 7.8
00:41 IST - crosses into Crater, apparent speed 0.82 degrees per minute, Mag 7.4
01:02 IST - crosses into Virgo, apparent speed 0.92 degrees per minute, Mag 7.0
01:05 IST - closest approach (approximately for India), apparent speed 0.92 degrees per minute, Mag 6.9
01:10 IST - close to beta Vir, apparent speed 0.91 degrees per minute, Mag 6.9
01:21 IST - crosses into Leo, apparent speed 0.87 degrees per minute, Mag 6.8
01:25 IST - close to beta Leo, apparent speed 0.84 degrees per minute, Mag 6.9
01:44 IST - crosses into Ursa Major, apparent speed 0.69 degrees per minute, Mag 7.2
02:33 IST - close to M109 galaxy, apparent speed 0.34 degrees per minute, Mag 8.4
03:24 IST - crosses into Draco, apparent speed 0.07 degrees per minute, Mag 9.3
04:57 IST crosses into Camelopardalis (headed directly towards Polaris), apparent speed 0.07 degrees per minute, Mag 10.5
Dawn break in Delhi

Astronomer Phill Plait will be doing live hangout during this event , you can catch him at Google Plus.

Tuesday, February 5, 2013

ISS Flyby 4th Feb 2013 Hyderabad

[caption id="" align="alignnone" width="1600"] ISS Flyby 4th feb 2013 , Hyderabad Photo by Joy[/caption]



Click the images to enlarge | Photo Credits :- Joy

ISS Flyby over Hyderabad on 04 Feb 2013 1930 hrs crossing Alpha Ceti and Aries . It was the brightest object in the sky with magnitude - 2.8 (Moon less night and Jupiter was magnitude -2.48 )

Chart from heavens-above.com

Thursday, January 31, 2013

Online slate board using HTML5 websocket and glassfish

My last post was about #AdoptAJSR initiative in which JUG Hyderabad is actively involved. Meanwhile I was trying to explore the Websocket API for Java (JSR 356) using glassfish 4 's latest build as reference implementation.

The idea was to use HTML5 websocket API in the front end to communicate with websocket endpoint created in the glassfish server.  To get started with I referred a  similar sample application blogged by Arun Gupta.

In this process I managed to log following bugs against the reference implementation

  1. Session object coming as the parameter in @WebSocketClose annotated method is null

  2. Session#isOpen method returns NoSuchMethodException

    To avoid this situation I have put the code "peer.getRemote().sendString("Ping");" in try block so that if it fails , it can gracefully remove that peer instead of ending up in exception.


My app lets you draw freehand lines in the browser and the same gets drawn in real time at every other user's browser who is accessing this app. If you refresh the page or a new user joins in then the existing state of the drawing is shown the browser.

To create such project one has to go maven way as of now. Following are the major steps you need to care about while creating this project

  1. Choose a Maven project in Netbeans


Maven1

Maven2

Maven3

2. The Websocket Java APIs could not be imported in the Java class unless the websocket library is searched in the repository and added to the project

maven4

maven5

The source code that I created can be accessed from Github. Feel free to download and give feedback or try to create your own version.

You can download the slate demo video and play in any flash enabled browser.

Play this in full screen to see the code.
Unable to display content. Adobe Flash is required.

Further features and future challenges

  1. Draw the existing lines slowly (Playback) when a new user joins in so that he gets the context. For this, one needs to block other users ( javaScript thread ) from drawing till the playback is over. or some way to buffer the new drawings till the JavaScript thread is free from playback.

  2. Different color and strokes

  3. Simultaneous drawing using webworkers.