-
  Q: "What's an easy quantitative feature you can extract from natural language text, such as corporate annual reports?"
  A: "The number of characters in a text string can be computed with just the builtin `len()` function. Word count might also be interesting, once you are familiar with the tokenizers in packages like SpaCy or nltk. You can also check for the presence or absenc e of particular keywords using `df['name'].apply(lambda s: keyword in s)`. Just replace `keyword` with a word you think might be interesting such as \"awesome\" or \"horrible\"."
-
  Q: "What is a good way to expand my network for a Data Science job search?"
  A: "Update your status with a link to a github repo for a project that you are proud of, a software package that you find useful, or an article about a data science concept that you liked. Your outreach communication should offer some information to your reader. And keep doing that every week, interacting with anyone that comments on your update. Also, prioritize contacts with recruiters. They will be able to advocate for you in the job market."
-
  Q: "What programming languages should I learn to get ready for a career in Data Science?"
  A: "It's best to master one general purpose programming language like python first. This will help you build a mental model of all the different data structures, algorithms, and programming patterns that you can then transfer over to almost any other language."
-
  Q: I can't resample a time series with duplicate time stamps.
  A: Try averaging all the values for duplicate timestamps first.
-
  Q: How can I create samples of a time series in a dataframe for 10 second intervals.
  A: "`df.timestamp.dt.round(freq='10S')`"
-
  Q: "I was wondering what is/are your data wrangling methods of choice? I find myself using `df.groupby()` just because it makes sense to me and goes with my way of thinking. I think it will take some experience to get used to melding and joining tabos or creating pivot tables and reindexing a table."
  A: "
  Yea, `groupby` is a powerful method. Stick with it. But there's one other tool you probably want to have in your bag of tricks and that's just the ability to iterate through the rows of a table and compute anything you want using if statements and math on all the columns. The pattern is:

  ```python
  new_feature = []
  for rownum, row in df.iterrows():
      new_feature.append(row['column_name'] * row['another_column_name'])
  df['new_feature_column_name'] = new_feature
  ```

  Can you think of any creative features you might generate using this approach on your data. It's expecially useful for latitude and longitude data and other geographic or time data.
  You can also use this approach to create a feature using data in another table (which is equivalent to join or merge). Basically once you understand the syntax of a for loop, you can do almost anything.
  "
-
  Q: "As a Data Scientist in industry do you use SQL in the real world?"
  A: "If you have a strong computer science background or teach yourself web development you can often learn how to avoid using SQL at all, by using ORM (Object Relation Mappings) in python or your preferred language. The advantage of that approach is that your brain can learn one programming language syntax very well. But as a beginning data scientist, without significant python and web development experience, you will probably have to use some SQL to retrieve the data you need to do Data Scientist. And even as an experienced Data Scientist who uses ORMs to retrieve and manipulate data, you will often need to use SQL to improve the efficiency of your queries. And some queries and database operations are not possible at all without an understanding of SQL."
-
  Q: "As a Data Scientist in industry what query language do you use to query a NoSQL database, such as a graph database or document store? I'm thinking of databases like Hadoop, ElasticSearch, MongoDB, Reddis, and Neo4J."
  A: "Neo4J uses Gremlin. NetworkX has a python API. Neo4J uses Cypher. ElasticSearch uses their own custom DSL (domain-specific language) based on JSON. You can find many other languages for NoSQL listed on the Wikipedia [NoSQL article](https://en.wikipedia.org/wiki/NoSQL)."
-
  Q: What is the best practice for selecting an index for my data in a DataFrame or database? Should I use the company name concatenated with the year as the index for financial data, or should I create a MultiIndex with company and year as the two levels of the index?
  A: In most cases its best to let pandas or your database management system create the index for you by simply leaving it empty and adding independent columns for all the variables in your dataset, like company name and year. This will allow you to join on any combination of the variables that you like. The one exception to this in Pandas is that if you often need to look up (retrieve) records by company name or year or both, it would be best to use them as a MultiIndex in addition to maintaining them as columns in your DataFrame of feature variables.
-
  Q: "How can I count the number of records for a particular group of data, like the `country` column in a pandas DataFrame?"
  A: "`[(country, len(group)) for country, group df.groupby('country')]"
-
  Q: I have blood alcohol levels from a breathalizer measured every 30 minutes. Subjects also carried a phone that recorded their x, y, z values based on accelerometer readings from the phone, measured approximately 50 milliseconds between samples. How can I join these tables to build a machine learning model to predict sobriety (blood alcohol level) based on accelerometer data.
  A: Try aggregating the accelerometer data by measuring the variance, standard deviation, min-max separation or some other scalar statistic over a few minutes preceeding the blood-alcohol measurement. You may need to do this with a manual for loop over the blood alcohol level measurements to add the acceleromater data to new columns of that table.
-
  Q: What is a CDF (Cumulative Distribution Function) used for?
  A: To estimate the probability or confidence interval for a range of values.
-
  Q: How can I read a large json file (more than 4 GB) without crashing my computer? It's the the yelp reviews dataset from Kaggle.
  A: "Try opening the file with the python open function. You'll need to set encoding to 'latin' for that particular Kaggle dataset. Then iterate through the file one line at a time (`for line in open(filename, encoding='latin'):`). Use `json.loads(line)` to read in a dictionary for each row of data and you can filter based on any of the dictionary keys that you like."
-
  Q: What is a confidence interval?
  A: A confidence interval is a range of values for a variable and the probability of a random sample falling within that range of values. Usually this is a range around the mean value in the dataset. This is because the mean value is usually the most likely value. And for a symmetric probability distribution, it is the center of the distribution. For example, for a large dataset of dice rolls (a pair of two die) the distribution is centered on a dice roll value of 7. For a pair of dice the 1/6 or 16.67% confidence interval is 7 +/- 0. The (5/36 + 6/36 + 5/36) or 4/9 or 44.44% confidence interval is 7 +/- 1. This is the same as saying you have a 44.44% chance of rolling a 6, 7 or 8 when you roll a pair of dice.
-
  Q: How is the gradient and gradient descent used in back propagation.
  A: Gradient descent is used to decide how much to adjust the weights on the last layer of the network first. But instead of adjusting the weights the full amount, the back propagation algorithm uses an algorithm to decide how much of the gradient to compensate for so that it can retain some error to propagate backward to the early layers. The back propagation algorithm is designed to avoid the problems of vanishing and exploding gradients.
-
  Q: What is the difference between stemming and lemmatization?
  A: A stemmer removes suffixes and prefixes from a word based on spelling rules. Lemmatization attempts to find the root of a word and can use a variety of approaches like WordNet or Word2Vect.
-
  Q: What is a good way to improve upon a Naive Bayes sentiment analysis model that used hand-tuned keywords?
  A: The most straightforward would be to use a TFIDFTransformer to filter out only those words with high df and low df and use all the other word fequencies as features in your model.
-
  Q: In an A/B test where you observe increased traffic for A when compared to the web traffic for B, what does a low P value mean?
  A: It means that the difference in web traffic that you observed for A relative to B is statistically significant and is not just the result of random chance or good luck.
-
  Q: When should I use a T-test vs a Z-test?
  A: It usually doesn't matter. When you have at least 10 samples to work with, both will give you very similar p-values. A T-test is slightly more conservative because it anticipates the possiblity of outliers. The T-distribution used to calcualte the T-statistic or T-test has fatter tails than the normal distribution which is used for the Z-score or the Z-test.
-
  Q: What is overfitting?
  A: When the accuracy on your training set is much better than for your test set. Test set accuracy is all you care about, because that helps you estimate how well your model will work in the real world. So having really high training set accuracy and overfitting is not necessarily a bad thing. Training set accuracy represents the best you could hope to do with that model and pipeline. So overfitting may suggest opportunities for improving your test set accuracy or other performance parameters.
-
  Q: How is unsupervised learning or clustering used in the real world?  #data-science
  A: Clustering (unsupervised learning) can sometimes be used to reduce dimensionality by creating an arbitrary class variable label to replace a high dimensional feature vector with. For example A natural language text field or image could be replaced with a label identifying a group of images or texts with similar content.
-
  Q: Is it OK for me to use external resources like stack overflow and google to complete the SQL exercises?
  A: Yes. But make sure you know how to change the query to do something different and still have a working SQL query. For instance try limiting the SELECT to fewer rows or a different set of rows within the same query. Or try joining a different pair of tables. Or try different aggregation functions like MIN and MAX. Or try different WHERE clauses like ">", "<", "<=" and LIKE queries.
-
  Q: I am unable to import pymc3 in my jupyter notebook. I installed it with pip from the command line.
  A: Try installing it withing the Jupyter notebook using `!pip install pymc3`.
-
  Q: How do I decide between all the different approaches to filling missing values, like `df = df.fillna(df.mean())` vs `df = df.fillna(0)`.
  A: This is a quesiton about choosing the best "hyperparameter" for you pipeline. The fillna method is a hyperparameter that should be recorded in your hyperparameter table or research notebook. The answer to these hyperparameter tuning questions is almost always that you must try each approach and compare the performance of each one to decide which is better. Each problem is different. Some methods work better in some situations than in others.
-
  Q: How can I experiment with different feature extraction approaches without recleaning my data with each new feature extraction approach that I experiment with in my hyperparameter tuning table?
  A: Try creating a data cleaning jupyter notebook or python script that is run only once. It's purpose is to save a csv with `df.to_csv('clean_data.csv', index=False)` at the end of the cleaning pipeline. All your subsequent hyperparameter tuning experiments should load that exact same file and use the exact same random seed to `train_test_split` the data.
-
  Q: How can I try different feature extraction approaches without rerunning my data cleaning with each new feature extraction approach that I attempt in my hyperparameter tuning table?
  A: Try creating a data cleaning jupyter notebook or python script that is run only once. It's purpose is to save a csv with `df.to_csv('clean_data.csv', index=False)` at the end of the cleaning pipeline. All your subsequent hyperparameter tuning experiments should load that exact same file and use the exact same random seed to `train_test_split` the data.
-
  Q: How do I know what features to extract from a natural language feature in my dataset?
  A: A TFIDFVectorizer object in scikit-learn can automatically extract a TFIDF vecto representation of your docouments (natural language strings). You can adjust the `min_df` and `max_df` arguments to filter out words that are likely to be less useful as model features based on their document frequency. Start with a min_df of 32 and max_df of 0.5, decrease min_df gradually while monitoring your RAM usage after the `.transform().to_dense().to_csv()` on your documents. You would like to use about 50% of your RAM when you load your entire TFIDF table (each row is a TFIDF vector) into RAM.
-
  Q: My models are getting the exact same accuracy whether I use an SVM or LogisticRegression. What can I do?
  A: First examine your training and test sets to ensure they have a sufficient number of diverse samples. You should plot the data on scatterplots, whenever possible, so you can see the individual records as dots and ensure you are covering the space of parameters you expect the model to deal with.
-
  Q: What is a good machine learning project dataset for someone interested in robotics?
  A: UC Irvine has a dataset for predicting sobriety from `(x, y, z)` position data. Any image segmentation or computer vision dataset would be applicable to robotics projects like self-driving cars.
-
  Q: What should I include in my data wrangling report the the song title and genre dataset?
  A: "Every data wrangling report should include the output of df.describe(include='all') for all tables you plan to use and an estimate for the counts of each kind of variable in your dataset: numeric, categorical, binary, ordinal, natural language, datetime, geographic, image, or audio."
-
  Q: How much text description should I include in my data storytelling report the the song title and genre dataset?
  A: Describe any interesting statistics you observed in the length of titles or most frequent words in the natural language fields. Your code and plots in a jupyter notebook will usually tell the story for you.
-
  Q: How can I reduce overfitting and improve test set accuracy of a LogisticRegression model in scikitlearn?
  A: Decrese the C value, this increases the regularization strength.
-
  Q: Imagine you had a dataset with number of bedrooms, bathrooms, square footage, roof type, exterior color, price, average monthly energy bill, and average monthly property tax. What would be some good hypotheses to test or machine learning target variables?
  A: You could predict home price from the structural information like bedrooms and bathrooms. You could predict the energy bill from the structural information in order to identify features that could be adjusted to reduce the energy consumption. You could even predict the exterior color in order to help people chose their paint that is most popular in their neighborhood for homes like theirs.
-
  Q: How is the pymc3 package used in the real world?
  A: "It's rarely used by developers or machine learning engineers in business, though it may be used by data scientists in academics or possibly finance. It has some helpful PDF plotting functions, but it's interface (API) is not pythonic: it relies on side effects, implicit behavior, and hidden functionality. Most python developers prefer to explicitly compute the statistics like p-value and confidence intervals that pymc3 provides."
-
  Q: "How important is it to understand the formulas used to perform a T-test and calculate P-value?"
  A: "It's very important to understand what a T-test is used for in the real world and how to interpret a low or high P-value. But it is not important at all to understand the formulas used to compute P-value or the assumptions about the data that were used to derive those formulas. I will make sure you understand the statistics principles you need to do well in an interview and at work in a Data Science roll."
-
  Q: "How can I visualize two tables of time series data on the same plot?"
  A: "`ax = sns.scatterplot(x='datetime', y='any_numerical_column_name', data=df1); ax = sns.scatterplot(x='datetime', y='any_numerical_column_name', data=df2, ax=ax)`"
-
  Q: "What does it mean for python code that claims not to use any 3rd party libraries?"
  A: "It means that the script does not import anything that had to be pip installed. Build in libraries like  `requests`, `collections`, `re`, and `itertools` are built into python and can be run on any python interpreter anywhere. Python scripts that import 3rd party libraries like `pandas`, `numpy`, `sklearn`, or `sqlalchemy` must install those packages before they can run the script."

