Showing posts from 2019

Deploying apps in Google Cloud Platform using GitHub Actions.

After GitHub announced action last year, they recently announced that it would support CI/CD free for the public repository as well. This is exciting because earlier most of the public repos were leveraging 3rd party Travis CI.  Travis CI is really simple for straight forward use cases, sometimes just one line. And also free for open source projects (Public repos on GitHub)

I had signed up for GitHub action beta program earlier and recently got the invite. After signing up you can see the new "Actions" tab in the top of all of your repositories

Earlier I blogged about how to deploy to google cloud from GitHub repository using Google Cloud Build. Here I wanted to use GitHub Actions instead to directly deploying a cloud function.

To get started one needs to have a .github/workflows directory at the root of the repo.Under the above directory, one can have multiple .yaml files to configure multiple workflows which can be run independently.The important sections of the action YAM…

Downloading And Storing Financial Data With Google Cloud Scheduler, Cloud Storage and BigQuery

Financial data from the stock market usually are available through APIs like Bloomberg API. Almost 12 years ago I worked for a company where they needed download and store historical data with the C SDK and also run a periodic cronjob to execute the same every day for that day's data (e.g. Day high, close, low, open prices and volumes). This was mostly run aftermarket when it was supposed to be available for that day. Ultimately data was stored in PostgreSQL and cater to many applications like Option Simulation (predicting the premium in upcoming days before option expiry using the volatility calculated by Black Scholes model ) and plotting bar charts (.NET or Adobe Flex desktop charts). The Simulation software was all written in PHP (in an object-oriented way) and some business users also used connectors from Microsoft Exel to fetch data from PostgreSQL directly.

While all these financial features were ahead of time, had all these done been done now, the obvious choice was proba…

We have a world chamipon !

Finally, we have a world champion and its England! It was the closest game (let alone in a world cup final) one could practically have. England built the team indeed played remarkably well since their elimination in the last world cup at the group stages. It definitely reminds me of India who lost in 2007 at the league stage and then ended up becoming the champions in 2011 at home. Both had started building a team for the long term which paid off.

There was a lot of talk about the final match being decided on the basis of number of boundaries hit and the overthrows off Ben stokes bat but the fact is that everything was under the rule and no one did anything intentionally to take advantage of it against the spirit of the game (unlike Ashwin waiting for runner to leave the crease marketing ) the rule were not new and both  teams were well aware. Like any other super over New Zealand did know their target is 16 and 15 belongs to England. It's not like after the Super over was tied I…

World cup 2019 so far.

Now that Bangladesh has scored 7 runs, we have our confirmed semi finalises in the cricket world cup 2019. There was a lot of calculations and jokes how Pakistan could make it to the semifinal on the basis of net run rate by beating Pakistan with 300+ runs. After all, there were so many similarities with the 1992 world cup,, how could Pakistan be eliminated in the league stage itself. Pakistan almost got eliminated in 1992 tough. Anyway as an Indian the 2nd best incentive after winning the world cup itself is beating Pakistan. Which we have been doing invariably right in every world cup encounter so far.

Personally, the history of the cricket world cup fascinates me more than anything else in the cricket. I did try to submit an article on it (Indias world cup campaign from 1975 to 1992) during 1996 world cup in my school magazine but it was rejected because it was not patriotic poetry or fictitious story. Anyway, now that we have internet and blogging social media platform we can wri…

Google Cloud Next '19

Courtesy Google Developers I got the opportunity to attend Google Cloud Next '19 in San Francisco last week. With more than 30 thousand attendees, It was a great experience to attend a conference of such a scale. I have heard it's only very few conferences like the Salesforce's Dream Force and Microsoft's Ignite are bigger than this. The venue was famous Moscone centre which is home of the largest tech conferences including GDC and Oracle Open World. In the past, it has also hosted Apple's WWDC and JavaOne for a long time which is also followed by developers all over the world.

The scale of the conference was so huge that, the price of the hotels in the city were surged up. Apart from the main set of Moscone buildings, some of the sessions were hosted in the theatres in Metreon and palace hotel. They released the mobile app and schedule of sessions much before the conference and one had to reserve the seat for some of the sessions in hot topics in Google cloud. I …

Automatically Deploying to Google Appengine from Github using Cloud Build

When it comes to CI/CD pipeline, Jenkins is one of the popular tools that a lot of software companies use. With google cloud build one can build and deploy their app right from source repository like GitHub to AppEngine. It's a pay as you go service. However the first 120 build-minutes per day is free, so one can very easily try for their side projects where the deployment frequency is not more than a few times a day. However its always a good idea to limit your billing account and have alerts on usage in case it goes over the free tier due to some build stuck and running indefinitely. Here we will take an example of a go based appengine standard project.

First, to have Cloud Build be able to deploy to appengine we need to give the service account it's going to use the roles that are required.

Find the project id

NUM=$(gcloud projects describe $PROJECT \ --format="value(projectNumber)") && echo ${NUM}
Derive the service account name from the project id and as…

Google Cloud Study Jams 2019, Hyderabad Week 3

After successful week 1 and week 2, it was time for week 3 of the Cloud Study Jam Hyderabad 2019. It was a full day event where participants were supposed to do labs from the Qwiklabs quest Baseline: Data, ML, AI with a lot of labs starting from Big Data services from GCP like BigQuery, CloudSQL, Dataproc to NLP APIs and Dataprep to ML Engine. More time was dedicated to this session on purpose. To make a data-driven decision we (GDG Hyderabad) had recently posted a poll and asked the community about the topics they are interested in.

Which topic would you love to learn from @googlecloud study jams? #StudyJam19#CloudStudyJam19#GDG#GoogleCloud#GoogleCloudFacilitator#GoogleCloudStudyJam#Hyderabad — GDG Hyderabad (@gdg_hyd) February 13, 2019

And as expected there were more participants in this session than last time. After talking to few participants it seemed a lot of them are already doing machine learning but they were looking for running it cloud to take full advantage of its elastic…

Google Cloud Study Jams 2019, Hyderabad Week 2

After the massive opening week of Cloud Study Jam last week. We had a great week 2 with follow up sessions on Google Cloud SDK, Appengine, CloudSQL, CloudFunction, Cloud Datastore etc. I was the primary instructor in this session, so I have been looking forward to this day for a while. Like last time the attendance again beat out expectation. The retention rate was beyond our imagination. It shows not only that the 1st week was really successful but also people are genuinely interested to learn about the cloud.

Kudos! Wonderful to witness great retention at @googlecloud Study Jam #Hyderabad 2019 - [Week 2 - Deploy & Develop] session. — GDG Hyderabad (@gdg_hyd) March 9, 2019 From the quick survey, it turned out most of them already completed the week 1's quest - GCP essentials or at least completed the parts which are pre-requisite for this session or just to get started with Google Cloud Platfor…

Google Cloud Study Jams 2019, Hyderabad Week 1

Last Saturday we had the 1st session of Google Cloud Study Jams in Hyderabad.

Week 1 Meetup

ThoughtWorks,, Hyderabad hosted us and they will be hosting the subsequent sessions lined up on next few Saturdays as well.
#Cloud#StudyJams kickstarted in style! Kudos to all those developers for taking their time out off a busy weekend to learn #GCP#CloudML#BigQuery and lot of interesting stuff over next few weeks.
Special thanks to @ThoughtWorks#Hyderabad team for being an amazing host for us!! — Uday Mannam (@UdayMannam) March 2, 2019
Thank you, everyone, for attending our first ever @googlecloud study jam in #Hyderabad in collaboration with @gcdc_hyd & @qwiklabs 😀. Looking forward to seeing you all for the Week 2 - #Deploy & #Develop#CloudStudyJams Photos: — GDG Hyderabad (@gdg_hyd) March 2, 2019
This session was all about introducing Study Jams to the developers and get started with the GCP e…

Profiling Java application in Google Appengine Flex

There are a lot of ways java applications can be profiled for memory and CPU to find memory leaks or any culprit block of code where the app spends most of the time. In Google AppEngine Flex the application code (jar or war) runs inside a docker container which in turn runs inside a VM dedicated a service/module. While Google Cloud Profile can be used for analysing live applications it only supports CPU profiling when it comes to Java in AppEngine flex. Memory profiling is supported for Java only in AppEngine standard and python and go in AppEngine flex. So as of now, one has to profile the memory in JVM manually.
SSH to the AppEngine instance If you have Editor role you can ssh to the specific appengine flex instance using gcloud utility. You can connect to the instance by clicking on the SSH button against the instance in the instances page in the appengine GCP console.
 SSH to the instance
$ gcloud --project "demoneil" app instances ssh "aef-default-20190225t133842-…

Container Registery and Minikube

In the previous post, we saw how to create a docker image out of a simple gradle/spring boot app and run it locally. In this post, we will push it to a container registry and use it to deploy it in a kubernetes cluster

Upload to Registry 

I created an account in Docker Hub and did a "docker login" in the command line with my credentials.

docker push neilghosh/gradle-spring-boot-app:latest
Note docker CLI bu default talks to Docker Hub, however one can log in to other container registries e.g. GCP's GCR or Azure's ACR by prefixing the image name with the fully qualified registry login URL.

Install and Setup Minikube
We can run Kubernetes locally using Minikube, this gives us the opportunity to develop/deploy/test quickly before we deploy into remote cluster available in GKE or AKS. Minikube uses a virtualization software like Virtual Box to deploy its containers

Download Virtual Box

Install Minikube
curl -Lo minikube…

Quick Microservice With Gradle Spring Boot Docker

A few years ago if you would have asked me to create a quick POC backend service with some rest APIs I would have just picked up NetBeans and created the REST APIs using jersey/glassfish (reference implementation of JSR 311 RESTful Web Services) from the UI itself. While the above is an example of creating REST APIs bottom-up from the Database layer, once can plug business logic by adding few more java classes or a service layer. Lately, I have been using Spring Boot a lot in production and it really makes the developer's life easy by not having to write a lot of boilerplate code. Annotations take care of Dependency injection, singleton, REST API mappings etc. It even has the embedded Jetty web server to run the service locally without really having to install a bulk of binary like Glassfish/Tomcat server and deploy them (comes integrated with Netbeans tough - one could right click…

I am now a Google Developers Expert !

I was recently recognised as a Google Developers Expert in Cloud. This is a program run by Google Developers to recognise developers of Google technologies as well as outstanding professionals in product strategy, UX/UI, marketing, growth hacking and monetization.

While this is a great hour I would definitely thank the local community leaders and Google DevRel folks for providing me with the opportunity over the years to get involved with the communities and nominating me for this program.

Google Cloud Platform is a public cloud where I have worked most professionally and personally in side projects. I was fortunate to discover this early on was just when it was just Google AppEngine with only python and datastore support. This was a big leap over finding a free shared Linux hosting with LAMP stack and cPanel where I also hosted manage my blog in WordPress. This was probably the only way I could host some server-side scripting / dynamic web page for free before AppEngine was there. Ev…

Analysing Google Cloud Platform Billing Data

No matter if you are a large enterprise or small startup, or even individual developers, you probably need to to know what you are paying for when it comes to cloud bill. in GCP you can export your billing data to Google BigQuery and then run queries on it to know where you are spending more which can help you control the cost you are incurring to run the infrastructure and app.

Daily spend
SELECT India_date, ROUND(SUM(cost)) cost FROM ( SELECT DATE(DATETIME(usage_end_time,"Asia/Kolkata")) India_date, cost FROM `YOUR_BILLING_EXPORT_TABLE` WHERE = 'YOUR_PROJECT_ID' AND _PARTITIONTIME >= TIMESTAMP("2019-01-16") AND _PARTITIONTIME <= TIMESTAMP("2019-01-20") ) GROUP BY India_date Order by India_date Result
GCP Cost per day, taking timezone into account.

 Cost By Services
    FROM (

Poco Phone F1 - Moving from Motorola to Mi

I have been using Motorola Android phones for a while now. Moto G, Moto X Play, Moto G 4 Plus, Moto G5 S Plus, Moto E and So on. They are not only considered to be the robust phone but also they run stock Android OS. It was part of Google and Nexus series for a while. However recently I gave up on them because of their camera. Even today they don't have a good phone with camera comparable with other phones in the same price range.

I bought the Poco Phone F1 recently and it has a really good camera comparable to the flagship phones like One Plus, Samsung Galaxy S* etc. Initially I was skeptical about the Xiomi phones because they were new, cheap and it is a Chinese company but I changed my opinion after hearing from friends who have been using the same Xiomi Mi phones since more than 2 years and have excellent photos from its camera for half of the price you pay for flagship phones. Poco was there best phone so far from the specification wise (probably except Mi Mix).

Here are the…

Yi 4K Action Camera

This is probably one of the first reviews I have ever written. You would find a lot of content about gadgets in youtube (with so many tech channels these days) but the only problem I have with that is I can not skim-through the video content just to see if there anything relevant that I need to know. I need to see/hear the full specifications and comparison what the video has to say. With this post (and possibly a series of posts with other gadgets that I own ) I am trying to have some notes that are usually not in the Amazon listing or manual or any overall review videos.

Yi 4K Action Camera one of the affordable action camera. Following are the observation.
The build quality is really good and quite compact.The time lapse works great, here is a sample. This is the most detailed version i.e. one frame per 0.5 seconds. For slow moving scenes like the cloud or sea, one can use even slower settings like one frame per 10 seconds and beyond.I also tested for the other extreme - slow motion…