Bala

GOPALAKRISHNAN

VP Innovation, Analytics

and Technology Architecture

THE WEATHER NETWORK / PELMOREX

Big Data: more than just a fair-weather friend

 

retour au sommaire

What is Pelmorex’s core business?

 

Pelmorex is one of the largest data and media organizations in Canada dedicated to the weather. It has large number of consumers - about 46 million users globally. From a content perspective, a lot of data is generated. Also, the users generate audience and behavioural data through interactions. Choosing to work for the firm gave me a good opportunity to further develop my expertise and learn about Big Data. It is a company where the data is very fertile, and as the Big Data revolution grows, we are only natural to capitalize on that moving forward.

 

The Big Data side of our business has become much more active over the last 2 years. When we looked at big players such as Google or Facebook, we realized that one of the biggest assets we had going forward was data. We produce a lot of content: weather forecasts and micro climate forecasts are generated every few minutes to every few hours. We have an editorial news team that write about the weather and climate change. The company has 24/7 television channel providing people with information about the weather and generating lots of video content and also live video streams, interactive apps on internet for over the top video devices like Apple TV, Android, TV, Roku etc. We combine all that with the potential mix of millions of users, who open our website and mobile applications numerous times a day, generating their own local weather photographs, videos, submitting information whenever there is weather activity or any news to report. This all represents a massive data asset that we can use in many different ways to both engage customers and create useful products for them.

 

 

What is your role as VP Innovation, Analytics and Technology Architecture?

 

My current role reports to the organization’s CIO and has three main functions.

The first is related to the amount of data we collect and generate. In order to process all the input raw data and make end consumer products available on multiple distribution platforms such as televisions, large multi-language websites serving several different countries, mobile apps on iOS and Android, iPads, Tablets, smart TV and OTT devices, we need to have the right Architecture in place. I lead this area to ensure we have the right technology systems in place, capable of supporting the various media platforms.

Then there’s thinking about how to mine this data effectively in order to gain insights we can then convert into new, useful projects. Our product is free to end-users, everything is monetized through advertising. We consequently want to use data in two ways - to derive useful consumer insights, so they find the content really engaging and to target them with the right advertising. Both perspectives rely on analytics.

Any company needs to innovate to stay current and useful to the consumer in today’s changing world, and to us innovation comes from combining the data we have with technological advancements. Today, people even access weather reports via smart watches, so it is vital we constantly innovate to remain leaders on different platforms. Computers are now more powerful, there are new ways of processing meteorological information, new advances have been made in the meteorological science field. Innovation is a combination of all this progress in technology, data and science. I coordinate or lead projects that bring those aspects together.

My team combines lots of these advancements to create new products and services, which they then prototype, test in the marketplace and present to the management team and key stake holders. That then makes it easier for us to decide each year which new consumer initiatives we are going to launch.

 

 

To whom do you report within the organization?

 

My current role involves working with four different types of people: data people, such as data scientists or data analysts; front and back end developers; a meteorological science team who are largely weather scientists studying algorithms; and finally my clients, who are my business partners in the company: Product leaders on consumer side and Advertising sales folks who monetize and generate revenue using the insights we have on the advertising side.

My company has realized that data represents a massive opportunity, so I am going to move to a new role that completely focuses on data - Managing Director of Data. That means I’ve progressed from combining multiple roles, to a highly specific senior role focusing on nothing but data. This role reports to the CEO. The CEA and board are definitely convinced of the importance of data - the fact we have so much data on what is happening in the marketplace was enough to convince everyone.

We still have to deliver a lot of positive results to prove we have made the right choice. We have already seen some promising results and early successes, but we are still 30-40% from being a totally data-driven organization.

 

 

Can you tell us about how data is used at Pelmorex? What kind of data do you collect?

 

We collect and produce weather data, primarily weather forecasting data such as what the weather will be like in the next few hours, in the next fortnight or over the next few months. We have specialized products like minute by minute forecast of rain and snow in next hours, long range weather outlook on calendar and seasonal forecasts. In itself, weather is a huge data set including observation stations, weather camera images, hourly forecast, radar, satellite imagery coming in from hundreds of thousands of locations around the world. 180 million forecasts are generated by our Forecasting engine for every kilometre in Canada, for instance. And we also draw on former weather patterns and climatological data, using 50 years of historical weather data for different places around Canada.

 

We also have articles and videos explaining both the in the moment weather data but also discussing topical issues such as climate change.

One of our biggest assets is user interaction. For instance, you could be looking at the forecast today in Paris, but you may be planning to fly to Toronto in 2 days’ time. We might notice when you use our app that you are starting to look at the weather in Toronto, allowing us to infer that you may be planning a trip. We can consequently bucket you into our business traveller segment for Toronto, allowing us to advise our partners to push relevant advertising deals on business hotels. From a content perspective, meanwhile, we might show you videos or articles related to Toronto. That is what we call behavioural audience data.

 

Another aspect relies on the fact that the weather is always consulted in a location sensitive context, related to where people are or where they are going. We are one of the services that asks people to turn on their GPS in order to use our application, and the acceptance rate is somewhere in the region of 75% to 80%, which is really high. This alone is a data set, as we get around 1 billion location records every month from different users, which we then use to create new consumer products.

In Toronto, there is a place called Toronto Island, which gets very busy in the summer. Based on location data, we can actually analyse the times at which traffic will be busy on the island and then proactively use that information to send push notifications or alerts to people living close to the island or who go there on a regular basis, telling them it’s the perfect weather to go there, but warning them to set off early as it’s likely to be crowded.

Our company philosophy is that “people use weather information to make decisions”, either when planning something or for safety reasons, and we can help by combining the audience, behavioural and location information we get and the unique content we have.

 

We have a lot of unique datasets, such as the surface and conditions of highways and roads, and the places where lightning strikes. We even have data related to weather and illness, such as where pollen is found in the summer, that we can use to help prevent allergies.

 

 

What kind of architecture do you use?

 

Our architecture is described as a hybrid. A lot of our audience data is collected using Google tools from Google Analytics, and we use Google Big Query, which is a Google Big Data Stack. For our users’ behavioural data, we use Microsoft Azure, HD Insights and Stream Analytics to capture real-time information. For predictive modelling and to detect correlations between the weather and other parameters we use a combination of Amazon Web services, Spark and some in-house clusters.

 

We use all the stacks, and our technological decisions are based on what is cost effective and what is the best option in terms of cost of data transport for a specific use case. For example, as our audience data is generated through Google Analytics, it is easier for us to transfer that data into Google Big Query, and then carry out our analyses there and connect the dots.

It saves us both money and time because the two systems are interconnected and we can rapidly deliver meaningful insights.

 

On the weather side of things, we run our meteorological algorithm to get better forecasts using in-house clusters, because both the algorithm and the science are highly proprietary. We use different types of models - American, Canadian and European - , so our meteorologists have built a model that blends up to 30 different models. Usually, most of them offer information covering a surface area of 15 to 50 kilometres. Our algorithm takes things down to a much finer level, blending multiple models and applies proprietary algorithms to produce weather forecasts for a specific postcode or neighbourhood covering up to one kilometre. For instance, I live close to the lake, so the general model may not be particularly relevant, as where I live is cooler than the surrounding area. Our algorithm takes that into account and produces a microclimate forecast, specific to the surrounding 1km.

We also create a system of virtual observations. There are real observation stations around the world with sensors in them, which detect the temperature, humidity etc.... but they are limited in number. To solve this problem and create more observation points, we use the forecast information for the next 15 minutes combined with radar information, which is more granular. Virtual observation is very accurate as it mainly relies on computer engineering processes to extrapolate data based on various data sources.

 

 

How successful is your Weather Predictive Analysis?

 

Prediction science was around well before Big Data came into play, but Big Data brought three new things:

  • much more input data;
  • much more processing, meaning we can   blend many more models;
  • much more granularity: in the past you would get the weather forecast for a city, but today a city can itself have several different microclimates.

Recently, Big Data has also allowed us to create a new algorithm called “when is it going to rain”, but down to minute by minute prediction from 30 minutes to 6 hours’ timeslots. That type of time precision for a specific microclimate is one of the areas in which Big Data has progressed, enabling us to create the precipitation start-stop algorithm. It is between 70 to 100% accurate, which is pretty impressive.

 

 

How do you connect all this data to the users of the applications?

 

Our company’s motto is: “Weather is there to serve consumers’ planning and safety needs”. We constantly strive to understand consumers, what their daily planning and safety needs are. We also try to identify which needs are not yet fully met. Our data teams examine these challenges and try to figure out what can be done to better address the various needs, attempting to find the right user experience by combining data, UX, product and technology. The precipitation start-stop is one of the problems that we identified as an important consumer need.

 

There are a lot of new products that can be generated by data itself. We have access to huge amounts of consumer data relating to both behaviour and location, and this data fuels new product concepts. Our typical prototyping cycle is about 3 weeks long; we try to fail quickly. Taking the product from the drawing board to the market - or not - typically takes around a couple of months to few months. For instance, the weather affects the mood of our users, so we are thinking of developing an application that allows people to give feedback on how they feel about the weather, asking users what their mood is in order to create a metric such as the mood of the day or the mood of the hour. This would be a totally crowdsourced product driven by its users.

 

 

What is the Roadmap of Innovation at Pelmorex?

 

In terms of Connected Things, it has already been multiplied by 100, and we expect the amount of data flowing in to be 100,000 times bigger. The more information we have, the better the predictions will be. Our challenge is to cope with all this information coming in and to manage it without having huge capital outlays. We also need to filter the noise - there is a lot of information coming in but lots of it can be misleading.

We can create very useful and granular micro climate forecasts, but at the end of the day that needs to translate into useful insights for each person’s specific context. How do we go about understanding the context where each person is standing? Understanding the types of decision, they are trying to make? For instance, if you are in an airport, looking at the weather forecast, you probably need to make a different decision than you would if you were standing on a soccer field with your children. The second challenge is consequently to provide consumers with the right product, using the location signal, user behaviour data and all the information available about them to understand and use the context, thereby delivering products that reflect each user’s context.

 

Finally, weather-related decision making for enterprises is another area that is opening up to us. Every business is affected by the weather. Business supply chains, sales activities and even product pricing can all change based on the weather conditions. This is something we plan to develop in the future.

 

 

Bala Gopalakrishnan has a background in technology and holds a Master’s in Computer Engineering from the University of Alabama in USA. Prior to joining Pelmorex, he worked in the technology and software development field. His first position was with Johnson Controls, a Fortune 100 company in developing hardware and software for computer-based controls industrial/building systems. He then joined a start-up, which grew into a larger organization, called Eutech Cybernetics, working in the IoT/Smart Cities technology integration sphere. Having completed his MBA in Canada, he was keen to work at the cross intersection of technology and business, inciting him to join Pelmorex, which is an innovative leader in bringing new products to consumers combining technology, science and creativity. He has been with the company for approximately 10 years in multiple roles, working on content, product and technology.