-
  Q: "What should a good presentation of a data science results look like?"
  A: "5-10 slides with one plot or diagram or picture per slide and 5-10 words per slide, with notes on the insight or purpose that you'd like to convey with each slide."
-
  Q: I have a table of song titles, genre, year-released, ratings, and numerical values like beats per minute, and song duration. What are some statistical inferences and statistical tests I can do to tell a data story?
  A: Any categorical variable, like year or genre can be used to create two different sets of data whose statistics can be compared. For example the mean `beats_per_minute` for 2010 might be compared to the mean `beats_per_minute` in 2000. You could then calculate confidence intervals using the Z-score for each of those distributions. You could even calculate a P-value using a T-test. It's almost certainly going to be very low if you have more than 100 songs or samples for each category.
-
  Q: Should I talk about the technical details of my work in my resume or linked in profile.
  A: Only when it contains words or concepts that are in job descriptions you plan to apply for. For example neuroscience or nuerons or "neur"anything because it's associated with neural nets and AI and data science by resume filters. However talking at length about "mitocondria calcium transport mechanisms" is probably not a good idea.
-
  Q: How do I deal with high dimensional one-hot encoded categorical variables from `pd.get_dummies()`?
  A: Use the same techniques you use with natural language tokens. There may be good word embeddings that will work for each of your categories (if they are natural language words). You can also do PCA on the one-hot vectors, as long as you combine multiple one hot vectors (for different categories) into the high dimensional data you are passing into PCA or TSNE.
-
  Q: What's a good problem for doing time series forecasting that employers will be interested in.
  A: Financial forecasting of time series like stock prices or Bitcoin prices based on Tweets is a popular project with Financial firms. And predicting web traffic like the Wikipedia traffic prediction project on Kaggle would be useful to almost any business with a website.
-
  Q: What are some good industries to learn about for data science jobs?
  A: Domain knowledge about an industryis not nearly as important as learning about the different datatypes that are common in the real world. Tabular data is the most common. Natural language data is everywhere and usually businesses have a lot of it that they don't know how to use. time series and geolocation data is also very common and requires a specific set of skills to get good at.
-
  Q: Is it important to select a machine learning or data science model based on the distribution plots and histograms of the data?
  A: No. For example, textbooks explaining Linear Regression will explain that it assumes your data is normally distributed. But Linear regressions work as well or better than many other models on skewed or non-normally distributed data. The onloy way to know whether a model will work well or not is to try it and do good cross validation or testing.
-
  Q: How can I record my hyperparameter tuning experiments in a machine learning pipeline without manually typing the results into a spreadsheet?
  A: "Create an empty list at the beginning of the script. Create a `for` loop that is iterating through a list of hyperparameters, like `for a in [.1, 1, 10]:`. Within the loop create a dictionary with all your hyperparameters and results, like `results = dict(alpha=a, test_acc=test_acc, train_acc=train_acc)`. Then you can append that `dict` to the list within the loop like: `results_list.append(results)`. After the loop that list can be converted to a DataFrame for plotting, sorting, and analysis: `df = pd.DataFrame(results_list)`."
-
  Q: I have a building dataset going back to 1970 with information like elevation of first floor and flood plane elevation combined with insurance claims data for flood events. How can I normalize for severity of flooding and value of the undamaged building before the flood?
  A: If you include all the variables you can measure in your model, the regression will automatically allocate the appropriate weight to each variable.
-
  Q: "What are the main questions I should ask about a dataset when chosing a good dataset for a project?"
  A: "1. How many rows (examples or samples) are there? 2. How many columns of data are there? 3. How much missing data is there in each of the columns? 4. What kind of data is in each column (numeric, categorical, natural language, datetime, geospatial, image, or sequence)?"
-
  Q: "Why would stop words be important in financial statements when predicting financial performance metrics like revenue."
  A: "Imagine the title of the CEO is listed as 'CEO and Chairman of the Board'. The word 'and' might be important to estimating any governance or misaligned incentives that could affect financial performance. It's better to let the statistics of the data drive the model performance rather than your own intuition, no matter how obvious you think a particular choice is, like filtering stopwords."
-
  Q: "When I calculate a 95% confidence interval with 1.96 * standard_error I get a smaller range than if I use norm.ppf(.95) * standard_error. Why?"
  A: "Because `norm.ppf(.95)` is calculating the one-sided z-score. The `norm.ppf` function is computing a probability using the inverse of the CDF (continuous probability distribition function) which starts at negative infinity when calculating a one-sided propbability. If you use `norm.ppf(0.975)` you will get a z-score much closer to 1.96 because it's computing the 2-sided distribution. You know that the normal distribution is symmetrical. It has the same on the left and right side."
-
  Q: "When will I be done with my capstone project?"
  A: "Once you have a hyperparameter table that experiments with all the models and parameters you want to learn about, you are done."
-
  Q: "Should a hypothesis be a statement or a question?"
  A: "Usually a hypothesis is a quantitative statement of a prediction about a relationship between two variables (quantities). The statement should be provable with data. For example for da, such as 'Homes in LA cost more than homes in San Diego.'"
-
  Q: "I am fitting a logistic regression on two different datasets and getting the exact same test set accuracy for both models. What can cause this?"
  A: "The data must not be exactly the same or it must be correlated in such a way that the model is converging to the exact same solution. Do `df.describe()` on both datasets to confirm that they are indeed different. Then try `df.corr()` to confirm that the new columns you added in the second dataset aren't highly correlated with the original dataset columns. Finally, try normalizing both datasets with the sklearn MinMaxScaler or StandardScaler to ensure that both models are finding the best solution to your data."
-
  Q: "When I try `df.dtype` it returns an AttributeError. How can I find the data type within a DataFrame?"
  A: "Try: `df.values.dtype`."
-
 Q: What data cleaning should I do when I suspect erroneous values in a dataset?
 A: First you should flag them as potentially erroneous in a new column with True's and False's aligned with the values you find suspect. You should then "correct" them, only if you are absolutely certain of what the correct value should be, otherwise you should leave them as is.
-
  Q: What is a polynomial regression?
  A: You can perform a polynomial regression in scikit-learn by training a linear regression on features that have been squared or raised to some exponent. You should also include the unsquared original features. This kind of model is useful when the relationship between your features and your target variable is nonlinear.
-
  Q: What does the F-value mean for an ordinary least squares multivariate linear regression?
  A: It is the ratio of the explained variance (variance of the predictions) divided by the unexplained variance (the variance in the residuals) for a linear regression. An F-value larger than 2 means that the model is statistically significant and is comparable to a P-value of less than 0.05.
-
  Q: What is F-value?
  A: "A measure of how statistically significant and reliable a multivariate linear regression model will be in the future. It is computed by dividing the explained variance (variance of the predictions) by the unexplained variance (variance of the residuals or errors). It is often used in conjunction with the R-squared correlation coefficient which measures the strength of the model in predicting the outcome or response variable."
-
  Q: How do I decide what kind of machine learning model to use for a particular application or dataset?
  A: The Scikit Learn documentation on ["Choosing the Right Estimator"](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html) is a great place to start. But in the end you will want to try as many models as you can, to gain experience and build up your own intuition about them.
-
  Q: What is a spacy `Doc.vector`?
  A: Spacy `Doc.vector` represents the average of the word2vec vectors for all the words in a document.
-
  Q: Difference between TFIDFVectorizer and TFIDFTransformer?
  A: A Transformer takes an input vector of continuous numerical data and transforms it. A TFIDFTransformer is used for a vector of word counts to transform that into TFIDF vectors by dividing by document frequency. Vectorizer takes string input and tokenizes it to count up the occurrences of words. A TFIDFVectorizer is a Vectorizer and a TFIDFTransformer combined, so it computes TFIDF vectors from strings.
-
  Q: I'm creating a time series forecasting model to predict whether Bitcoin price goes up or down using historical daily prices. What is the appropriate target variable value when there is no change in the price?
  A: It is very rare that a price will not change at all, so you can just put it in either category (up or down) using a `>=` or `<=` comparison to compute your target variable.
-
  Q: What is a good dataset that contains both datetime and geographic information so I can get experience with building spatiotemporal models?
  A: The Melbourne housing market dataset has latitude and longitude, postal code, address, datetime, and price information, along with many other parameters. And it has about 50k records, so it would be a great one to start with.
-
  Q: What is a good dataset for getting started in Data Science.
  A: The [Melbourne housing dataset](https://www.kaggle.com/anthonypino/melbourne-housing-market?select=Melbourne_housing_FULL.csv) is a good one to start with. It has spatiotemporal data with latitude and longitude in addition to dates and prices. There are 34 columns of house characteristics to use to predict price.
-
  Q: Help me select a good dataset for my next project. I want to work on a problem that is not as difficult so that I can have a model that has some use in the real world.
  A: Looking at the accuracy as a measure of a model's usefulness in the real world is not helpful. It's possible to make billions of dollars off a model that can achieve 50.1% accuracy at predicting weather the financial markets are going to be up or down. And a model that predicts car accident severity with even modest accuracy can save 100s or even thousands of lives per year if you interpret the coefficients of the model and legistlatures acted on them.
-
  Q: I have a dataset that contains ski resort names, lift ticket prices, number of lifts of the various types, and the opening and closing dates for the resorts. Another resort wants to know whether they should install a new lift and whether their revenue will make up for the added expense. What's a good problem statement?
  A: Choose a _target variable_ from your dataset. This is a variable that would be interesting to management if they could predict it based on the other _feature variables_ in your dataset. Then you can propose a problem statement that predicts that variable and suggests actions to affect it based on the model coefficients for each of the feature variables. So your problem statement might be adding an additional lift may increase lift ticket prices and increase the number of days a resort stays open. You could test this hypothesis with a multivariate regression on ticket price vs the other variables in your dataset.
-
  Q: Can I combine the IMDB dataset with the Netflix dataset?
  A: What column or combination of columns could you use as the key in each table to join on? what about the movie title plus the year? How can you resolve ambiguous titles? How can you deal with different spellings and capitalizations of titles in each database?
-
  Q: What is a good plot to include in my final report?
  A: Scatter plots are always helpful. Residual plots are very important. Hyper parameters and their affect on training and testset accuracy are often revealing.
-
  Q: What is a good place to find state of the art (SoTA) benchmark datasets?
  A: The site [paperswithcode.com](http://paperswithcode.com) has a button at the top where you can explore "State of the Art" papers in various areas of deep learning. Look for github links where there are often instructions for downloading the associated datasets.
-
  Q: What is QSAR data?
  A: QSAR is (Quantitative Structure–Activity Relationship) chemistry and biology data typically used in a regression analysis to predict chemical compounds and their biological effects.
-
  Q: What is a good plot to use in place of a stacked bar chart for a binary categorical variable (the colors of the bars) and an ordinal variable with 5 possible values (the height of the bars).
  A: Two overlapping normalized histograms or bar charts with the ordinal variable as the horizontal axis and the normalized count of the ordinal as the height of the bars. There should be two independent set of bars in the chart, both rising up from the bottom for both of the 2 values (colors) in the categorical variable. Either the two colors should be side by side or should be transparent (have an `alpha` color value of around .5).
-
  Q: Is a `LogisticRegression` more likely or less likely to overfit to your data when compared to a `DecisionTree`?
  A: A `LogisticRegression` will be less likely to overfit than a `DecisionTree` in most situations.
-
  Q: Is a `RandomForest` with `num_trees=3` more likely or less likely to overfit to your data when compared to a `DecisionTree` model?
  A: A `RandomForest` will be less likely to overfit than a `DecisionTree`.
-
  Q: What advantages might a human have over a system like AutoML or DataRobot?
  A: Human intuition and domain understanding is required to understand the results and their implication for business decisions in the real world. In addition a Data Scientist's intuition can help guide the hyperparameter search as well as the feature selection process.
-
  Q: A Lasso linear regression doesn't seem to perform as well as some other models on most problems. What is it good for?
  A: A Lasso model is good for helping to identify the most important features for a linear model. It is mostly used for feature selection, when there are many independent features with a linear influence on the target variable.
-
  Q: In my Lasso and SGDRegressor models, increasing `max_iter` doesn't help. Why?
  A: Perhaps the actual number of iterations is below the max_iter limit that you have set. For instance, if your fitted `Lasso` model is called `lasso`  you can see how many iterations it ran to achieve the fit with `lasso.n_iter`. Any value of `max_iter` above this number is unlikely to affect your model fit at all.
-
  Q: What does the F-statistic mean?
  A: It gives you the T-statistic of your model relative to an intercept-only (mean) model. It's almost always extremely large if you have a large number of samples. It is rarely useful in non-academic settings.
-
  Q: How can I manually set the y-axis scale?
  A: Use `plt.ylim((y_min, y_max))`. For example if you want your plot to range from 0 to 5 on the vertical axis you'd use `plt.ylim((0, 5))`
-
  Q: What's a good label for my vertical axis when I've taken the home prices in dollars and divided them by 1000000?
  A: I'd suggest 'Home Price ($M)'.
-
  Q: Is it necessary to know command line `git`?
  A: It helps when you're interacting with developers on a shared software project.
-
  Q: Data coming from my Android users seems to be different from my iPhone users. Should I build two different models for each user set?
  A: No. A better approach is to create a new binary feature variable called "is_android" or "is_iphone" and add it to your model. This will allow your model to generalize some features between the two platforms while specializing the model on those features where an Android and iPhone differ.
-
  Q: The Melbourne real estate price dataset on Kaggle mentions that we should try to predict the downturn in the market from the provided dataset, but the only date information is the "YearBuilt". There is no "YearSold" or "DateSold" information.
  A: This dataset is not sufficient for answering that time series problem statement posed on the Kaggle website. But you can use it to create a predictive model on price.
-
  Q: Is the global happiness index dataset on Kaggle a good data science project to start with?
  A: No. It doesn't have very many records and all of the columns are likely directly used to compute the happiness index.
-
  Q: Should I use the one-sided or the 2-sided T-test for a hypothesis like training will increase an employee's likelihood of achieving 80% or more of the KPI metrics?
  A: One-sided, because you're specifying a direction  of the change in your hypothesis.
-
  Q: My training set accuracy is often much worse than my test set accuracy.
  A: Make sure you have a representative sample of your dataset in your test set. Ideally you should have at least as many eamples in your test set as you have features. If you have a "wide" dataset, like for a TFIDF table for an NLP problem, then you should have at least 25% of your samples in your test set.
-
  Q: Every once in a while I get a negative accuracy or correlation score for a Ridge or Lasso regression. Why is this?
  A: Some models, like RandomForest and Lasso are stochastic, which means they use a random number generator to make decisions within the algorithm. So to produce repeatable results with these algorithms you should set the random seed before training them or doing your train test split with `np.random.seed(your_favorite_number)`.
-
  Q: What is the purpose of `train_test_split`? Why is it good to separate your data into two different random samples of data?
  A: The model is fit to the training set and the test set is reserved for testing only and is not ever "seen" by the model during the training phase. The test set accuracy score is an estimate of how well your model will work in the real world. The training set accuracy is an estimate of how good the model could possibly be if you were able to find the best hyperparameters for your pipeline.
-
  Q: "My Jupyter notebook for training an RNN seems to hang after 10 minutes or so."
  A: "Check your memory usage using a tool like the Linux application `htop`. If your SWAP is being used or most of your RAM is used up then close all Chrome tabs that you don't need, and halt all Jupyter notebooks.  Restart the Jupyter notebook before trying again. Keep an eye on your RAM and use training set data generator parameters like BUFFER_SIZE and BATCH_SIZE to reduce the memory footprint of your model."
-
  Q: How do I calculate precision from the confusion matrix?
  A: Calculate the true positives with `tp = ((y_pred == y_true) & y_pred).sum()`. Calculate the true negatives with `tp = ((y_pred == y_true) & ~y_pred).sum()` and the false positives with `tp = ((y_pred != y_true) & y_pred).sum()`. Finally, calculate precision with `precision = tp / (tp + fp)`.
-
  Q: "How can I delete security credentials and large data files from my git repository history?"
  A: "Check out the `bfg` package: `conda install bfg`."
-
  Q: "My model for the Melbourne housing dataset is getting poor accuracy predicting price from the the continuous variables like square footage and number of bedrooms and lat/lon. What can I do?"
  A: "You can try some feature engineering on the latitude and longitude data by calculating the distance to the mean latitude and longitude and adding that as a new feature. You can also calculate the distance to any other landmarks that seem like them might be useful for predicting the home price."
-
  Q: "I'd like to do sentiment analysis on Amazon reviews by scraping the reviews from Amazon.com."
  A: "That's a bad idea. I'd suggest you first learn data science before learning how to scrape data from websites. Amazon reviews are available in several large database dumps online. There's no need to learn how to scrape. Scraping is difficult and scraping an Amazon.com page is probably one of the most difficult scraping problems."
-
  Q: Is it a problem if all my flood damage prediciton dataset was collected from buildings that had claims over the past 100 years?
  A: Yes, your dataset has significant sample bias. You may want to join it with a dataset that contains buildings that have not ever experienced a flood.
-
  Q: The data science process sounds like you are treating data the same way a Deep Learning algorithm would. You don't need to know what the variables mean.
  A: Exactly, you just need to extract a wide variety of features from your dataset and let the machine find patterns between your extracted features and the target variable.
-
  Q: What's a good way to explore the json string returned by a web API?
  A: Use `json.load` or `json.loads` to convert the string to a python object like a list of nested dictionaries. Then examine that python object like you would examine another other list or dictionary. Look at a few "records" in the list. Checkout the keys and values for one of the records in a list of dictionaries. And visualize in your mind or on paper the nested structure of any nested dictionaries.
-
  Q: I'd like to do a classification problem for my next data science project.
  A: Every regression problem can be turned into a classification problem by selecting thresholds to divide the dataset into classes.
-
  Q: I'd like to do an image recognition problem for my next data science project.
  A: I'd recommend the MNIST dataset. It's a handwritten digit recognition problem with a lot of good tutorials online. And it doesn't necessarily require deep learning.
-
  Q: What does gradient boasting accomplish? What is it's loss function?
  A: Gradient boosting attempts to normalize and randomize the residual statistics. So that's the loss function, the magnitude of the residual error and the mismatch between the residual PDF or histogram relative to the normal distribution.
-
  Q: Can SVM model a nonlinear relationship between the features and the target or do I need to generate polynomial features for that?
  A: The kernel trick should take care of the nonlinearities in your features to help create a model.
-
  Q: What is a good problem to work on for the IMDB dataset? I don't want to build a recommendation engine.
  A: What about predicting box office receipts divided by budget? Or just predicting budget and box office receipts individually. This would allow you to provide insights to producers, investors, and even actors that they'd be very interested in.
-
  Q: What is a target variable?
  A: It is the variable you are trying to predict. It's usually called `y` because it is often plotted on the vertical y-axis. The feature variables are often called `x` or `x1, x2, x3, ...` because they are often plotted on the horizontal x-axis of a plot. So a 1 variable linear regression will often be visualized as a line passing through a scatter plot with the target variable on the y-axis and the feature variable on the x-axis.
-
  Q: What is the Agile process applied to Data Science?
  A: It requires you to be ruthless with your time and focus 100% on getting bare minimum possible product that has some small value to you in the minimum amount of time. For Data Science this means a model and a report about the data that provides some increased understanding of it. For example if your dataset has a lot of missing values but has a few columns of data that are complete, then train your model on only those few features. Get a training and testset accuracy number to create a single row in your hyperparameter tuning table. Then put that table into a report (preferably a Jupyter notebook) with some short description of what you learned from that model.
-
  Q: Plotting the number of reviews along the y axis and the product price on the vertical axis it seems that price is inversely proporational to the log of the number of reviews. So I transformed my `num_reviews` with the log function to create a `log_num_reviews` feature, but this contains `-inf` values and fillna does not remove them. what should I do?
  A: First try to figure out what might be causing the `np.log` function to output `-inf` values. Try rely large values and really small values as well as really large and small negative values; such as `np.log(1e10), np.log(1e-10), np.log(0), np.log(1), np.log(-1), np.log(-1e10)`. From that you should see that for products that have zero reviews you are getting a `-inf` for your `log_num_reviews` feature. You can find these in your dataset with `mask = df['log_num_reviews'] < 0`. And then you can fill those entries with a value that makes sense, like 0 or -1 or perhaps even `df['log_num_reviews'][mask] = -np.log(1)`.
-
  Q: What are the pros and cons of having 5-folds for cross-validation vs 1-fold?
  A: 5-folds will run 5-times slower but it will create a better estimate of the accuracy of the model in the real world. It will also give you a confidence interval on your accuracy number. 1-fold is usually better for most problems because it allows you to explore 5-times more hyperparameters, increasing the speed with which you can find the most accurate model.
