Uber is committed to delivering safer and more reliable transportation across our global markets.
Interesting, no? I'm a stupid security researcher. For the moment, the keywords are: train, airport, city, geo fence... Do you see where we are going?
Digital health lawyer Jodi Daniel answers questions about what people can do to control how our medical and health data is used, as smart watches, fitness trackers and other devices gain popularity.
When I talk to people about machine learning on phones and devices I often get asked “What’s the killer application?“.
Hadley Wickham had the great suggestion of analyzing a Tidy Tuesday dataset. Tidy Tuesday is a fantastic project run by the R for Data Science online learning community (especially Thomas Mock) that releases an interesting dataset each week.
After almost three years at Jobr/Monster, I have decided to leave to pursue a different opportunity. This gave me the chance to interview and discover new companies while practicing my technical chops and learning the inner workings of different businesses and their utilization of data science.
DeepScale is constantly looking for ways to boost the performance of our object detection models. In this post, I’ll discuss one project we launched towards that end and the unintuitive discovery we made along the way.
Data-Driven Documents (D3) is a JavaScript library for building powerful graphics to communicate information in datasets. It is also fair to say that for many, myself included, it has a non-standard approach to building the graphics. Often the learning curve can feel steep.
This is a blog about the CRAM file format for storing DNA sequencing data. There is a lot of misinformation about CRAM out there. I'll be charitable and say it's accidental, but even accidental wrongs sometimes need correction.
Data and analytics professionals seem to be at the center of the next big race for talent. In 2015, there was a surplus of people with data science skills. Now there’s a significant shortage. By 2020, IBM expects broader demand for data and analytics talent to reach 2.7 million positions in the U.
I joined LinkedIn about six years ago at a particularly interesting time. We were just beginning to run up against the limits of our monolithic, centralized database and needed to start the transition to a portfolio of specialized distributed systems.
Data is ubiquitous — but sometimes it can be hard to see the forest for the trees, as it were. Many companies of various sizes believe they have to collect their own data to see benefits from big data analytics, but it’s simply not true.
I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. I wasn’t promoted or assigned to this new role. Instead, Facebook came to realize that the work we were doing transcended classic business intelligence.
This post is adapted from the blog of LeadGenius, a Priceonomics customer. Does your company have interesting data or insights? Become a Priceonomics customer.
That’s the question behind the new Chrome extension Data Selfie.
Five years ago, a team of researchers from Google announced a remarkable achievement in one of the world’s top scientific journals, Nature. Without needing the results of a single medical check-up, they were nevertheless able to track the spread of influenza across the US.
Introduction As I was browsing the web and catching up on some sites I visit periodically, I found a cool article from Tom Hayden about using Amazon Elastic Map Reduce (EMR) and mrjob in order to compute some statistics on win/loss ratios for chess games he downloaded from the millionbase archive,
Machine learning is a branch in computer science that studies the design of algorithms that can learn. Typical tasks are concept learning, function learning or “predictive modeling”, clustering and finding predictive patterns.
In the interest of getting back into writing, I want to break the seal with a simple “what have I been up to and thinking about lately” style post. Hopefully future topics will be more focused and frequent. For the past year, I have been working on data and analytics at GitHub.
For a company like Slack that strives to be as data-driven as possible, understanding how our users use our product is essential. We knew when we started building this system that we would need flexibility in choosing the tools to process and analyze our data.
Edit: Hey guys! This has proved to be one of the most popular articles on the site, so I’ve created a supplemental download on the 5 biggest statistics mistakes beginners make and how to avoid them.
Knowledge workers need greater visual literacy than they used to, because so much data—and so many ideas—are now presented graphically. But few of us have been taught data-visualization skills.
Let’s start with Facebook’s Surveillance Machine, by Zeynep Tufekci in last Monday’s New York Times. Among other things (all correct), Zeynep explains that “Facebook makes money, in other words, by profiling us and then selling our attention to advertisers, political actors and others.
“What is the relationship like between your team and the data scientists?” This is, without a doubt, the question I’m most frequently asked when conducting interviews for data platform engineers.
People are creating, sharing, and storing data at a faster rate than at any other time in history.
The ability of statistics to accurately represent the world is declining. In its wake, a new age of big data controlled by private companies is taking over – and putting democracy in peril by In theory, statistics should help settle arguments.
First things first, what is big data about? Control.
There are three elements to our "big data" efforts, or unhyped normal data efforts: Data Collection, Data Reporting, and Data Analysis. We are all aware that the best companies in the world have an optimal DC-DR-DA allocation when it comes to time/money/people: 15%-20%-65%.
You’ve probably heard about GDPR. The new European data protection regulation that applies practically to everyone. Especially if you are working in a big company, it’s most likely that there’s already a process for getting your systems in compliance with the regulation.
This article was originally published on October 26, 2016 and updated with new projects on 30th May, 2018. Data science projects offer you a promising way to kick-start your career in this field.
We’ve entered the age of big data, in which more and more companies are seeing the value and importance of data in many different areas of their business, from market and customer research, to internal sales figures and HR analytics.
Today’s developers often hear about leveraging machine learning algorithms in order to build more intelligent applications, but many don’t know where to start.
Over the past 7 years, Netflix streaming has expanded from thousands of members watching occasionally to millions of members watching over two billion hours every month.
Raw data is boring and it’s difficult to make sense of it in its natural form. Add visualization to it and you get something that everybody can easily digest.
You don’t need to be a seasoned data scientist or have a degree in graphic design in order to create incredible data visualisations. It has become a lot simpler to mine your data and interpret your insights in an engaging, attractive, and most importantly easy to understand way.
The more experienced I become as a data scientist, the more convinced I am that data engineering is one of the most critical and foundational skills in any data scientist’s toolkit. I find this to be true for both evaluating project or job opportunities and scaling one’s work on the job.
Data Science, Machine Learning, Big Data Analytics, Cognitive Computing …. well all of us have been avalanched with articles, skills demand info graph’s and point of views on these topics (yawn!). One thing is for sure; you cannot become a data scientist overnight.
In preparing this talk I decided to check out the data landscape, since I hadn't seen it for a while. The terminology around Big Data is surprisingly bucolic. Data flows through streams into the data lake, or else it's captured in logs.
Engineering teams face a common challenge when building software: they eventually need to redesign the data models they use to support clean abstractions and more complex features.
The goal of this tutorial is to introduce the steps for building an interactive visualization of geospatial data. To do this, we will use a dataset from a Kaggle competition to build a data visualization that shows the distribution of mobile phone users in China.
Data science is an exciting, fast-moving field to become involved in. There’s no shortage of demand for talented, analytically-minded individuals. Companies of all sizes are hiring data scientists, and the role provides real value across a wide range of industries and applications.
Data systems have mostly focused on the passive storage of data. Phrases like “data warehouse” or “data lake” or even the ubiquitous “data store” all evoke places data goes to sit.
This post is part of a series covering the exercises from Andrew Ng's machine learning class on Coursera. The original code, exercise text, and data files for this post are available here. One of the pivotal moments in my professional development this year came when I discovered Coursera.
(Sorry about the length! At some point in the distant past, this was supposed to be a short blog post. If you like, you can skip straight to the demo section to get a sense of what this article is about.) Embarrassingly, most of my app development to date has been confined to local devices.
Data is essential to us at Airbnb. We characterize data as the voice of our users at scale. Thus, data science plays the role of an interpreter — we use data and statistics to understand our users and translate it to a voice that people or machines can understand.
Big Data, Data Sciences, and Predictive Analytics are the talk of the town and it doesn’t matter which town you are referring to, it’s everywhere, from the White House hiring DJ Patil as the first chief data scientist to the United Nations using predictive analytics to forecast bombings on scho
Many conversations about data and analytics (D&A) start by focusing on technology. Having the right tools is critically important, but too often executives overlook or underestimate the significance of the people and organizational components required to build a successful D&A function.
How can organizations leverage data as a strategic asset? Data comes at a high price. Businesses must pay for data collection and cleansing, hosting and maintenance, salaries of data engineers, data scientists and analysts, risk of breach and so on. The line items add up.
AN OIL refinery is an industrial cathedral, a place of power, drama and dark recesses: ornate cracking towers its gothic pinnacles, flaring gas its stained glass, the stench of hydrocarbons its heady incense.
More than a million people have now used our Wolfram|Alpha Personal Analytics for Facebook. And as part of our latest update, in addition to collecting some anonymized statistics, we launched a Data Donor program that allows people to contribute detailed data to us for research purposes.
At 9.24pm (and one second) on the night of Wednesday 18 December 2013, from the second arrondissement of Paris, I wrote “Hello!” to my first ever Tinder match. Since that day I’ve fired up the app 920 times and matched with 870 different people.
It feels good to be a data geek in 2017. Last year, we asked “Is Big Data Still a Thing?”, observing that since Big Data is largely “plumbing”, it has been subject to enterprise adoption cycles that are much slower than the hype cycle.
Five years ago—in February 2012—an article in the New York Times’ Sunday Review heralded the arrival of a new epoch in human affairs: “The Age of Big Data.
“Learn to code.” Around the world, it has been a familiar refrain. And the world has duly learned. Time was that the standard in-school computer education was little more than word processing, spreadsheets and some basic programming.
Data Science is an ever-growing field, there are numerous tools & techniques to remember. It is not possible for anyone to remember all the functions, operations and formulas of each concept. That’s why we have cheat sheets.
What shall we make? This talk is about what’s coming next for us as designers. What’s the work ahead for us, and what’s our role and responsibility in this future? Designing for what’s next is what my studio Big Medium does.
This blogpost is an excerpt of Springboard's free guide to data science jobs and originally appeared on the Springboard blog. Most data scientists use a combination of skills every day, some of which they have taught themselves on the job or otherwise. They also come from various backgrounds.
This is the first post in a series covering Yelp's real-time streaming data infrastructure.
Data preparation, cleaning, pre-processing, cleansing, wrangling. Whatever term you choose, they refer to a roughly related set of pre-modeling data activities in the machine learning, data mining, and data science communities. ...
Hi! I’m Jose Portilla and I’m an instructor on Udemy with over 250,000 students enrolled across various courses on Python for Data Science and Machine Learning, R Programming for Data Science, Python for Big Data, and many more.
To understand visual patterns within the dataset quickly and efficiently, we worked with artist Kyle McDonald to overlay thousands of drawings from around the world. This helped us create composite images and identify trends in each nation, as well across all nations.
What does big data know about you? Quite a lot.
Dear Lifehacker, I've been hearing more and more about "big data." What is it, and is it something I should be worried about? Is this another way companies harvest my data and sell it? Dear Bewitched by Buzzwords, "Big data" is the latest tech industry buzz-phrase.
In this post in the microservices series we will study how to manage inter-service dependencies and how to deal with implicit interfaces in the form of data sharing. We will also extend our existing examples from the microservices series to show some of the concepts in this post.
Service-Oriented Architecture has a well-deserved reputation amongst Ruby and Rails developers as a solid approach to easing painful growth by extracting concerns from large applications. These new, smaller services typically still use Rails or Sinatra, and use JSON to communicate over HTTP.
In a tech startup industry that loves its shiny new objects, the term “Big Data” is in the unenviable position of sounding increasingly “3 years ago”. While Hadoop was created in 2006, interest in the concept of “Big Data” reached fever pitch sometime between 2011 and 2014.
Guest blog post by Rubens Zimbres, PhD. This article brings images from my work modeling with Mathematica, my experience as a Business Analyst and also my doctorate lessons.
On June 17, 2015, I celebrated my two year #Twitterversary @Twitter. Looking back, the Data Science (short for DS) landscape at Twitter has shifted quite a bit:
My sister is a senior undergraduate majoring in sociology. She just landed an awesome analyst job for next semester and was told she'll be using some R in the course of her work.
Does your data have a purpose? If not, you’re spinning your wheels. Here’s how to discover one and then translate it into action.
As the big data analytics market rapidly expands to include mainstream customers, which technologies are most in demand and promise the most growth potential? The answers can be found in TechRadar: Big Data, Q1 2016, a new Forrester Research report evaluating the maturi
The story of how data scientists became sexy is mostly the story of the coupling of the mature discipline of statistics with a very young one–computer science.
Misusing statistics is one of the most powerful ways to lie. Normally, we teach you how to avoid misinterpreting statistics, but knowing how numbers are manipulated can help you spot when it happens.
Every month, Segment collects, transforms and routes over 50 billion API calls to hundreds of different business-critical applications. We’ve come a long way from the early days, where my co-founders and I were running just a handful of instances.
Demand for skilled data scientists continues to be sky-high, with IBM recently predicting that there will be a 28% increase in the number of employed data scientists in the next two years.
Our new Keystone data pipeline went live in December of 2015. In this article, we talk about the evolution of Netflix’s data pipeline over the years. This is the first of a series of articles about the new Keystone data pipeline. Netflix is a data-driven company.
In the last few years I spent a significant time with reading books about Data Science. I found these 7 books the best. These together are a very valuable source of learning the basics. It drives you through everything, you need to know.
The more than 1.35 billion people who use Facebook on an ongoing basis rely on a seamless, “always on” site performance.
When Edward Snowden blew the lid off of the NSA’s mass surveillance program, he also revealed the extent of the government’s smartphone location tracking records.
Eric Ma has a PhD in Biological Engineering from MIT, and was an Insight Health Data Fellow in Summer 2017. Eric has since joined the Scientific Data Analysis team at Novartis, where he gets to pursue his dream role of doing data science in the context of the life sciences.
Daniel (not his real name) was a VP human resource manager at a Fortune 500 company. I asked him whether he had collected any data that could provide him with insights into systematic patterns. “I made sure we get exit interviews done with every single employee who is leaving us,” he replied.
A year ago, I dropped out of one of the best computer science programs in Canada. I started creating my own data science master’s program using online resources. I realized that I could learn everything I needed through edX, Coursera, and Udacity instead.
Recently we shared the techniques we used to save more than a million dollars annually on our AWS bill.
A year ago, I dropped out of one of the best computer science programs in Canada. I started creating my own data science master’s program using online resources. I realized that I could learn everything I needed through edX, Coursera, and Udacity instead.
On the cultural ideology of Big Data. “What science becomes in any historical era depends on what we make of it” —Sandra Harding, Whose Science? Whose Knowledge? (1991)
While working on my statistical analysis of 142 million Reddit submissions last year, I had a surprising amount of trouble settings things up. It took a few hours to download the 40+ gigabytes of compressed data, and another few hours to parse the data and store in a local database.
I joined Booking.com as a data scientist about two and a half years back, straight after a 3 years consulting gig in Dubai. Moving from consulting to a pure data science role was a big shift in my career and in hindsight I’m very happy I made that choice.
We love data, big and small and we are always on the lookout for interesting datasets. Over the last two years, the BigML team has compiled a long list of sources of data that anyone can use.
FREQUENT visitors to the Hustler Club, a gentlemen’s entertainment venue in New York, could not have known that they would become part of a debate about anonymity in the era of “big data”.
Exploratory data analysis (EDA) is an important pillar of data science, a critical step required to complete every project regardless of the domain or the type of data you are working with.
Imagine if a company’s three highly valued data scientists can happily work together without duplicating each other’s efforts and can easily call up the ingredients and results of each other’s previous work. That day has come.
You need analytics. I’m very confident of that, because today, everyone needs analytics. Not just product, not just marketing, not just finance… sales, fulfillment, everyone at a startup needs analytics today.
Alarmed that decades of crucial climate measurements could vanish under a hostile Trump administration, scientists have begun a feverish attempt to copy reams of government data onto independent servers in hopes of safeguarding it from any political interference.