-
  q: "How to use chi squared test in feature selection?"
  a: 'Personally I find it hard to know ahead of time what the right threshold for chi**2 test is. Instead I use automatic dimension reduction techniques like PCA, regularization, and hyperparameter tuning (grid search). These require less "judgement" on my part. I let the data and algorithm make the decision for me.'
-
  q: "What's the benefits of using log loss vs accuracy? And usually where log loss is used? (do we just treat for example 5 % quantile as outliers?)"
  a: "There are numerous cost/loss functions you can use, or you can make up your own. Almost all of them will give you the same answer to your optimization/fitting problem. But some will be better or worse at telling you how good your model is at predicting the things you care about. Sometimes you care about the standard error, sometimes you care about the log of the computer standard error, sometimes accuracy is more important to the success of your project, sometimes precision, sometimes recall, sometimes F1 or AUC. What do you think the agile approach would be to answering your question? Is there a way to use my answer to 1 to come up with an approach to question 2? Is there a way to ask your \"boss\" or customer which one matters most to them? What is the measure of units that most businesses measure success or lose (cost ) with? What are the best \"units\" for business decisions?"
-
  q: "How to detect outliers in a data set and how to handle them?"
  a: "Like other pipeline design challenges, I do outlier filtering in an agile way. Can you guess what that is? Spoiler alert... Outliers can be discarded automatically during hyperparameter tuning, or even within the model itself (random forest or xg-boost)"
-
  q: "Can I use .corr() or pierson correlation on a binary or categorical variable?"
  a: "Correlation is a mathematical statistic. It can be calculated on any numerical value where that numerical value contains information about the real world (integer ordinal, binary, float, or bool, but not a categorical variable until it has been one-hot encoded). What do we do with all categorical features to create numerical values that work in a machine learning pipeline? Hint: correlation is equivalent to linear regression."
-
  q: "Can I use .corr() or pierson correlation on an integer feature?"
  a: "think deeply about how an integer is different from a float. Is 2.0 Twitter likes the same thing mathematically as 2 Twitter likes ?  30.0 vs 30 years old ? What about 5.0 stars vs 5 stars? What about toaster oven low med high vs a numerical toaster knob with 0, 1, 2 or 0.0 1.0 2.0 ? What about word frequency of 0/1 vs TF-IDF of 0.0/1.0 vs Boolean word presence value of False/True ?"
-
  q: "Can I use .corr() or pierson correlation on an integer or binary feature?"
  a: "Each situation is different. You have to think about the \"physics\" of something to know what what the best numerical representation is. We can go through some examples to help you learn how to think about numerical value types like binary, int, float, boolean, categorical, ordinal, and continuous things in the real world."
-
  Q: "How can I get more confidence in how the time series predictions will work on real data in the future? "
  A: "Implement an expanding window validation and model training pipeline that trains 100s of models using the last day as the validation example."
-
  Q: "how to check the correlation b/t two variables between an integer and a continuous value? For example how many people likes a comment on twitter vs the age of user?"
  A: "Ordinals (integers) can be treated just like a continuous value throught a machine learning pipeline, including the correlation coefficient calculations."
-
  Q: "how to check the correlation b/t two variables between an integer and a boolean value? For example whether a user is an engineer or not. "
  A: "Booleans (True/False) can be treated just like a continuous value if they are convert to integer 0/1 values or floating point 0.0/1.0 values. They'll work throughout a machine learning pipeline, including the correlation coefficient calculations."
-
  Q: "Is pearson correlation coefficient work for both for booleans, ordinals (integers)?"
  A: "Yes. Booleans and ordinals are cast to floats (continuous values) and should work fine in any correlation coefficient or any other statistics calculation."
-
  Q: "What is a Markhov Chain model?"
  A: "A model of state transitions (sequences like time series) where the next state depends only on the current state and not any of the previous states. Example: predictive text that only looks at the last token you typed (or the \"<start>\" token) to predict the next token."
-
  Q: "What is a Markhov Chain Monte Carlo model?"
  A: "A Markov Chain that estimates the distribution for the next state by sampling historical data or a simulation. Example: predictive text that only looks at the last token you typed (or the \"<start>\" token) to predict the next token and estimates the distribution of possible next tokens based on historical data for all the texts you (or people like you) have typed in the past."
-
  Q: "What is a static model or distribution?"
  A: "A probability distribution or model whose underlying statistics or behavior do not change over time."
-
  Q: "What is a deterministic model?"
  A: "A model or algorithm that has a closed form solution (can be solved by evaluating a sequence of mathematical expressions) without relying on random sampling or other stochastic (semi-random) algorithms. A stochastic model or algorithm should converge to the same solution to a given problem (for the same dataset) regardless of when or where it is trained."
-
  Q: "What is a nondeterministic model?"
  A: "A model or algorithm that has no closed form solution and relies on stochastic or random sampling techniques to achieve a solution.$"
-
  Q: "Give examples of deterministic and nondeterministic models."
  A: "Stochastic: KNN, K-Means, Stochastic Gradient Descent? Neural Networks with random dropout or random initialization of weights, Random Forest, Decision Tree. Deterministic: Linear Regression, PCA, SVD, Logistic Regression, Polynomial Regression, Stochastic Gradient Descent?"
-
  Q: "What are the characteristics you look for in a Data Science or Machine Learning model?"
  A: "Training time/complexity per data sample and feature (dimensionality). Inference time/complexity. Data efficiency. Number of parameters (degrees of freedom). Linearity/nonlinearity of the solution. Nonlinearity/complexity of the generated model. Classifier or regressor. Supervised or unsupervised. Stochastic or deterministic."
-
  Q: "What is a leverage plot and what should I look for?"
  A: "I don't know."
-
  Q: "What is a residual plot and what should I look for?"
  A: "you want to look for any patterns or anomalies in the data. If the residuals are not normally and uniformly distrubuted, that's an opportunity for you to do additional cleaning, like removing outliers from your training set (but not your test set), filling in missing or incorrect values. If there's some clear curvature to the data then that's an opportunity to do some more feature engineering, like squaring or rooting or taking the log of all your features to create new features."
-
  Q: "What is a quantile plot and what should I look for?"
  A: "I'm not sure, but it seems to show the normal statistics (z-score) variation across various values of your target variable. It seems to plot the standard deviation and mean of your predictions in a rolling window across a sorted list of your target variable (like home price). So it reveals oportunities for additional feature engineering on the \"tails\" or edges of the plot, where you may have fewer examples and the model may need to include nonlinearities to deal with _edge cases_."
-
  Q: "What is False Discovery Rate?"
  A: "It's a way to adjust your P-Value or T-test or Z-test or X^2 test to account for the number of experiments you've run, then number of times you've compared your P-value to your 5% threshold (typically). If you only do one experiment you can be 95% confident that your results are going to be repeatable by others, if your P-value is below 5%. But if you do 2 experiments, an only one passes the p-value threshold test, then you can only be 91% confident. So to keep your confidence high, you need to tighten your P-value threshold based on the number of tests you've performed on a given dataset."
-
  Q: "What is the Kernel Trick?"
  A: "It's when an ML algorithm like SVM transforms the feature vectors using a linear or nonlinear kernel function. For example there may be a  linear rotation and scaling matrix that unwinds and flattens a spiral classification dataset (there's an example in playground.tensorflow.org)."
-
  Q: "And what does it mean when someone says that an algorithm like SVM doesn't actually transform the data with the kernel?"
  A: "The kernel can be selected and optimized within the cost function calculation of a SVM or SVC, so it's applied to the loss rather than to the higher dimensional feature vectors to save computational effort (I think)."
-
  Q: "Where can I find some good DS FAQs or anwers to common questions?"
  A: "In addition to this knowledgequest chatbot, you can try places like Springboard.com's community Kahn Academy and Kaggle.com. For deeper questions you should do some searches on scholar.google.com."
-
  Q: "Recruiter contacted me about jobs in Houston for DS, Sr DS and Data Analytics. The data analytics position looks like the work I already do, but it's a bit advanced. What should I do?"
  A: "Apply for all of them, and read the job descriptions with an eye for the experiences you've had on Springboard and at work with each of the tools, languages, challenges in those job descriptions. Take notes on how you used the tool, or solved a problem at work, whenever you have an interesting problem or tool or learn something from the other DSs or developers."
-
  Q: "The job descripotion mentions Spark, what should I do?"
  A: "Make sure you know what Spark is and think about all the places you've used it on Springboard or at work. What problems is it good at? Paralellizable machine learning pipelines that need to scale to more data than fits into RAM. The employer won't expect you to be able to code up a Spark pipeline during a code interview. But they will ask you to analyze a problem and describe how Spark might be helpful. "
-
  Q: "The job descripotion mentions ETL/ELT, what should I do?"
  A: "What does ETL stand for? \"Extract Transform Load.\" What does ELT mean? \"Extract Load Transform.\" Are they different in some way? Which acronym do you prefer? Which one makes more sense based on what your experience with ETL? Why? I prefer \"ELT\" even though it isn't as common as ETL. ELT is the order that I typically have to do things during the data cleaning and visualizaiton process. But in a large database, you may have to do a lot of the transformations incrementally on only portions of the DB or individual records, loading them one at a time and transforming them. So both acronyms make sense. Make sure you know what the hiring manager means by the terms Extract, Transform and Load terms and think about all the places you've encountered similar problems and tools at work or on Springboard. The hiring manager and team may ask about any experience you have with ETL and any insights you have about it, like how do you deal with missing values? How can you automate some of the most labor intensive parts of ETL? What are the challenges of ETL? What are some good ETL tools?"
-
  Q: "Why does GridSearchCV take so long? What's wrong with my code?"
  A: "GridSearchCV trains 3 different models on the data, by default, because it uses KFolds cross-validation with K=3. In addition, your implementation of your `create_model()` function trains and validates a Keras neural net model every time you call it. So the GridSearchCV is retraining and revalidating your model 3 times, for a total of 6 trainings.  Never use the black-box functions that do the work of a data scientist, like hyperparameter optimization or cross-validation. Training and evaluating model performance usually requires a lot of computation and memory and time, and you want to minimize that time by inserting your DS intuition into the optimization process. If you do want to automate a grid search use `itertools.product()` to manually iterate through your hyperparameter combinations and evaluate your model performance."
-
  Q: "What's the right way to do hyperparameter optimization?"
  A: "Bayesean search rather than grid search, or human-intuition-driven manual search. If you don't have probability distributions in mind that you want to test, just assume that the prior is a continuous distribution across a range of values or a list of discrete values. Check out the `hyperopt` package on github and pypi. If you can't figure out how to implement bayesean search, then just do a random sort of your grid search product(): `sorted(product(param1_list, param2_list, ...))`. And if you really want to speed things up, Start with random search for the values that minimize the training and inference time for your models. For parameters that increase the training time significantly, only try them rarely, and only with one parameter at a time."
-
  Q: "What are replicants in the context of bootstrapping?"
  A: "I don't know but I imagine they are the duplicates that occur due to random sampling with replacement."
-
  Q: "What kinds of problems are there in the inferential statistic exercises?"
  A: "Problems that indicate your understanding of Z-score, T-tests and other statistical measures of how likely it is that a statistic, like mean or sum, is significantly different in one group (the treatment vs the control)."
-
  Q: "What is the difference between Data Wrangling, Data Storytelling, and EDA (Exploratory Data Analysis)?"
  A: "What does data wrangling mean to you? Loading, cleaning, and organizing your data into a form suitable for visualization and modeling. Generally that means consolidating your data into a single table. What does data story telling mean to you? It's building a story about the real world that the data is telling you. You need to have done EDA and data wrangnling first so you have plots and statistics about your data that can help you come up with those ELI5 descriptions of what's going on in the real world to cause that data to appear in your database. What are people clicking on a particular button? What are men generally taller and denser, on average, than women? Those are examples of questions that data story telling might answer. What does EDA mean to you? Exploratory Data Analysis is the plotting and statistics calculations you do on your data to get a better understanding of what's going on in the real world to create those data entries. You want to look for patterns and correlations between features to see if they make sense or contradict your intuitions. These patterns will suggest models and statistical tests that you can use to build a Data Science pipeline and a data story."
-
  Q: "Are these plots useful or should I remove them?"
  A: "What are you trying to convey to the reader? What do you want them to get out of them? What do they tell you? Is that piece of information obvious at first glance? You reader will likely only look at the plot for a few seconds. In a bar plot, the bars next to each other should always have a relationship to each other that is interesting, since those will be compared to each other. If you have more than 10 bars on a single plot, they will likely hide the important pairs that you want your reader to look at, so cluster or group them more agressively. And never ever use raw counts on a bar plot or histogram, always us percent or normalized count so that your plot shows probability and allows your reader to compare values from different histograms with different total counts (like the cohorts in your income survey data)."
-
  Q: "The conclusion to my capstone starts with \"The above plots show that feature importance are driving model performance.\" What do you think?"
  A: "I'm not sure I understand what you are trying to say with that. Are you saying that the features with high feature importances values from the random forest model statistics are important to the model's decision tree branches? That's a circular statement that doesn't say anything new. Talk about the real world affects of your features on your target, income level. Talk about the fact that relationship status questions can have both a postiive and a negative impact on income. \"married with wife\" relationship status is highly positively correlated with income, whereas \"married with children\" relationship status is highly negatively correlated with income. And talk about how education level is not highly correlated with income, above a high school diploma. Those are interesting, meaningful, specific conclusions you can share with your reader."
-
  Q: "Which penalty function should I use for logistic regression, l1 or l2. My dataset has the heights and weights of hospital patients and the target variable is gender."
  A: "What do you think is the right way to chose between two different possible settings or values for a hyperparameter, like the regularization penalty metric for logistic regression? Try it, right? Whichever one improves your performance on the test set is the **right** one, right? And for this particular model your degrees of freedom is only 2 (`len(np.array(model.coef_).flatten())`) and the number of examples is presumably much larger than 2, probably 100s or even 1000s. So theres almost no change your 2 DOF logistic regression can overfit. So it doesn't matter what C value (inverse regularization strength) you chose and which penalty metric you choose."
-
  Q: "What is the difference between l1 and l2 penalty or distance? How does each one affect the regularization penalty in a logistic regression."
  A: "l2 penalty is crows flies distance. L1 penalty is manhattan distance. The 1 and 2 come from the exponent on each feature/dimension value that is applied before summing up the individual dimension distances to compute the distance. So a manhattan (l1) distance metric is usually better because a drop in an individual dimension coefficent in your model will have a greater overall impact on the penalty than for l2. Consider the diagonal of a 1x1x1 cube for a 3 DOF linear regression model with slopes of 1, 1, and 1, vs 1, 0, and 0. The first model would have an l2 penalty of sqrt(3)=1.7 and the second, much simpler model, would have an l2 penalty of 1. That's a drop of 42%. For manhattan (l1) distance the penalty would drop from 3 to 1, gives a drop of 66%."
-
  Q: Which penalty function should I use for logistic regression, l1 or l2. My dataset has the heights and weights of hospital patients and the target variable is gender.
  A: What do you think is the right way to chose between two different possible settings or values for a hyperparameter, like the regularization penalty metric for logistic regression? Try it, right? Whichever one improves your performance on the test set is the **right** one, right? And for this particular model your degrees of freedom is only 2 (`len(np.array(model.coef_).flatten())`) and the number of examples is presumably much larger than 2, probably 100s or even 1000s. So theres almost no change your 2 DOF logistic regression can overfit. So it doesn't matter what C value (inverse regularization strength) you chose and which penalty metric you choose.
-
  Q: What is the difference between l1 and l2 penalty or distance? How does each one affect the regularization penalty in a logistic regression.
  A: How do you chose any hyperparameter? What does l1 penalty mean? What does l2 penalty/distance mean?
-
  Q_teacher: "If you're given a dataset with 10 columns 1M rows of numbers and the 10th column is the target you are trying to predict. What do you do to get started with data exploration?"
  A_student: "df.describe(), df.info(), assume any low cardinality integers are categorical (get_dummies). Scatter plot of each feature vs the target. Linear regression, if no overfitting, random forest, if overfitting add regularization or feature reduction."
-
  Q: How does one decide what to do next with a 2-layer 128-cell LSTM network that is getting 95% training set accuracy and 91% test set accuracy for sentiment analysis? Does it have high variance or high bias?
  A: I don't like using those terms what's a better way to describe how your model is doing? (Overfitting vs underfitting)  You model is overfitting so you need to increase the random dropout and reduce the number of layers or the number of neurons in each layer. In general the only way to decide what to do next is to try it. And record your answers in a hyper-parameter tuning log or table. And record each new model as a separate python file or notebook so your can reproduce the results of previous models. Especially when it takes a long time to train.
-
  Q: should I get more data to reduce overfitting?
  A: getting more data is usually the harder thing to do so you should do it last. Instead try to make your model smarter by increasing regularization and reducing the number of features or doing feature reduction like PCA.
-
  Q: Is scaling and normalization useful for a Linear Regression? What about SVM?
  A: No, in general vastly different scales do not affect the performance of most ML models like linear regression. However a Support Vector Classifier and Support Vector Regressor do tend to benefit from normalized scaling of the data (for example `sklearn.preprocessing.MinMaxScaler()`). This is because an SVM optimizes the margin between target values/class, and normalizing your feature values ensures that the margin is balanced across the features. For example, the margin/separation between the class boundary for a feature ranging over +/- 100 would be 100x further away from the "true" class boundary than for a feature ranging between +/- 1, if feature normalization or scaling were not used.
-
  Q: My tensorflow+CUDA installation is not recognizing my GPU when I train a model. What should I do?
  A: What is your OS, CUDA, python, and tensorflow version. Mine works for CUDA 9.0 on Ubuntu 18.04 with python 3.6 and tensorflow 2.0. In addition, it may help to set the global environment variable `export CUDA_AVAILABLEGPUS="0"` if you have 1 GPU or `export CUDA_AVAILABLEGPUS="0,1"` if you have 2 GPUs.
-
  Q: Is the movie review dataset a good one?
  A: Yes.
-
  Q: "SQL is not applicable to what I want to do. Do I need to do these exercises?"
  A: "I don't use SQL much in my everyday work as a data scientist and machine learning engineer. However as a backend software engineer it will be a critical skill. And I've had many Data Science interviews where I asked or was asked SQL questions. Even people who only want to use Tableau to explore and visualize data often have to use SQL. One way around this is to use an ORM, like `Django` or `sqlalchemy`."
-
  Q: "Is SQL required for DS work?"
  A: "No, but it's required to get through many interviews."
-
  Q: "The inferential statistics exercises are hard."
  A: "Inferential statistics is all about using classical Data Science statistical approaches to create basic models to predict a target varialbe. And example of an inferential statistics model would be if you tried to predict gender based on only the height and weight of people. You could take the mean height and weight of the women in your training set, and likewise for the men, and then assume that whichever centroid your test examples are closest to can be used to assign them a predicted gender label. Obviously this model has flaws and a machine learning approach would involve creating additional features, like perhaps dividing the height by the weight and then finding the optimal threshold on this estimate of BMI (body mass index) to predict the gender (whichever threshold gives you the highest accuracy, precision, or recall, whatever you care about most). "
-
  Q: "How can a testing engineer position interview (asked about DS applied to test engineering) be parlayed into a Data Science position."
  A: "Once you have the job as a test engineer, you can show your colleagues how to predicting and diagnose issues with hardware or software (whatever you're testing) using machine learning."
-
  Q: "I'm in sales and marketing. How can I apply AI to my work?"
  A: "You can use ML (a form of AI) to predict customer behavior, like favorable or unfavorable response to a marketing campaign or the sales generated by a particular promotion. You could even build a chatbot or a generative model to perform that advertisement for you. Google is a leader in AI research and their focus is on advertising and marketing, mostly through the AI that is part of their search engine and ad-words program."
-
  Q: "What are some ways to learn about marketing, and marketing techniques that apply ML."
  A: "There are numerous blog posts and academic papers on applying ML to marketing and advertisement. Search for \"advertising marketing machine learning\" on DuckDuckGo.com, toolbox.google.com/datasetsearch, kaggle.com/datasets,  and scholar.google.com."
-
  Q_teacher: "What is a good way to plot data that has widely varying scale like the frequency or document frequency for the 50 most frequent words in my documents?"
  A_student: "I don't know, maybe zoom in on one part of the data? Oh I know, how about a log scale plot so that all the values have equal visibility."
-
  Q: "What values for min_df and max_df I should you use for the `sklearn` `TfidfVectorizer()` ?"
  A: "What does df mean? What does min_df do? What does max_df do? How does it help a model? What is the purpose. How does df vary across the words in your documents? How many documents would be affected if you increased the min_df by one? How much would your vocabulary size be reduced? How much would this help your model? What if you decreased max_df by 1 or by 1% (.01), how many documents would be affected? How much would this help?"
-
  Q: "Is it a good data science project to build a smart search tool for GEO and Environmental dataset like oil pipeline locations, whale and marine species migration data, farming data, etc."
  A: "No. Just gathering up the metadata about those datasets would be very difficult and time consuming. The best project ideas are not ideas that I generate on my own, they are ideas that are suggested by a dataset a table with numbers in it."
-
  Q: "What is a good dataset to practice joining datasets or tables?"
  A: "The conventional approach is to find an existing data science problems like predicting home prices and predicting crime in Chicago and then join them to create additional features for both problems. The crime data joined on the latitude and longitude or street address in both tables could be useful as a feature in predicting home prices. And the home prices and number of stories and other features of homes might be useful in predicting crime rates if joined with the crime table on the street address or geolocation of the crime reports."
-
  Q: "What about joining the Kaggle competition on predicting the dish name based on the ingredient lists in recipes, with another table from grocery stores about their stock of those ingredients. Is that a good Data Science project?"
  A: "Is it going to be easy to obtain a reqpresentative sampling of grocery store stock and prices from across the united states? It's not. So it is better to let the datasets suggest a project rather than letting you mind dream up hypothetical datasets that might be hard to obtain. For example can you think of a way to use the FDA.gov website which has CSV files of food items and their nutritional content? What if you joined it on the recipe dataset from Kaggle? What would your target variable be? (any of the nuttitional components available on the FDA databse, like sugar or fat or calories or vitamin C) Would it be supervised or unsupervised? (unsupervised) Use toolbox.google.com/datasetsearch and kaggle.com/datasets to help you come up with a dataset (or pair of datasets) and a problem statement."
-
  Q: "For hyperparameter tuning, feature selection in particular, if I add a new feature to my model, how do I know if it helped or hurt. "
  A: "If you see an improvement on your training set accuracy but your test set accuracy is degraded then the feature likely doesn't contain useful information for the model, so it can be skipped."
-
  Q: "For hyperparameter tuning, feature selection in particular, should I train a model based on a single feature at a time before training on combinations? "
  A: "It will be more efficient to simply accumulate more and more features by adding features one at a time and not training on them each independently. If you don't see an improvement in the training set or the test set accuracy, then you can likely skip it. Or you can keep it in and wait until your test set accuracy is degraded before your ignore features."
-
  Q: "For hyperparameter tuning, should I record the model coefficients using something like `model.save()` with every new combination of features or hyperparameters that I train it on?"
  A: "Not if the model is relatively fast to train/fit.  Because you've recorded your hyperparameters in a hyperparameter tuning log or table you can always recreate any of the models that worked well by refitting it using those hyperparameters and the training set you used the first time.  If the model is complicated with a lot of coefficients and takes a long time to train you can save a \"checkpoint\" whenever you see an improvement in your test set accuracy, so you don't have to try to recreate it later."
-
  Q: "What are the characteristics you look for in a Data Science or Machine Learning model?"
  A: "Training time/complexity per data sample and feature (dimensionality). Inference time/complexity. Data efficiency. Number of parameters (degrees of freedom). Linearity/nonlinearity of the solution. Nonlinearity/complexity of the generated model. Classifier or regressor. Supervised or unsupervised. Stochastic or deterministic."
  R: "Springboard 2019 DSC curriculum and Phoenix"
-
  Q: "What makes a good dataset?"
  A: "A large number of labeled samples is important to make the problem straightforward. high dimensionality can make the problem harder. Examine the relationship between possible target variables and the other features to make sure the target is something that would be interesting to \"predict\" if you only had access to the features."
  R: "Sprinboard 2019 student Ali"
-
  Q: "Should I look for something I'm interested in, or something that would be a good project?"
  A: "Both. Start with the dataset, like at the google dataset search tool. Look for interesting datasets where the data is arranged in tables that you can easily import and manipulate. CSVs with numerical values are usually the easiest. Think about the dimensionality of the features and the number of samples/examples/rows and whether you think the problem will be solvable. Examine the relationship between possible target variables and the other features to make sure the target is something that would be interesting to \"predict\" if you only had access to the features."
  R: "Sprinboard 2019 DSC student Ali"
-
  Q: "How can I view the sparse matrix output of a `vectorizer.transform()` method, like sklearn.CountVectorizer().transform()"
  A: "`doc_vectors = pd.DataFrame(sparse_matrix, columns=vectorizer.get_feature_names())`"
  R: "Springboard 2019 student Karolina"
-
  Q: Library usage survey .corr(), how do I visualize in a way that makes sense.
  A: Set the diag() of your correlation matrix to 0 so that it doesn't swamp the plot. Also display the correlations as a single column at a time with `C=df.corr(); C['colname'].sort_values().round(2)`. Also, in your heatmap, use seaborn and set the axis labels to the column names with `ax.set_xaxislabels(C.columns)`
-
  Q: I don't have a time series so that doesn't apply.
  A: Don't you have a survey submission datetime? That makes it a time series.
-
  Q: What are some features you can extract from a datetime?
  A: Is it a Holiday, is it summer, is it the weekend, is it Sunday, is it December, is it morning, is it lunch hour, etc.  Can you think of some more?
-
  Q: How do I know whether 12 LSTM cells is enough. I acheieved 91% sentiment classification accuracy with 128x2 LSTM cells and I achieved 90% accuracy with 12x1 cells.
  A: The only way to find out is to try. Use Daniel C Bennet's intuition pump "make mistakes." That's the best way to learn and develop an intuition.
-
  Q: What does a hyperparameter tuning table look like?
  A: It has a column for each hyperparameter that you changed and a row for each model that you trained. You should also have a column for the test set accuracy or loss. This is what you care about the most. You can also have the training set accuracy and maybe the test set accuracy at epoch 10, or some other arbitrary point in your training.
-
  Q: "What kind of accuracy should I expect for sentiment analysis."
  A: "92% accuracy is probably the upper limit for a large clean dataset like the Yelp restaurant reviews dataset and a sophisticated LSTM model. On other datasets 80% or even 75% may be the best you can achieve. Sentiment is a very gross target variable that can contain a lot of noise. Above 90% accuracy, you are probably starting to fit to noise. A human cannot likely achieve this accuracy, so a machine that does is superhuman, despite not having common sense knowledge or empathy."
-
  Q: "What is the state of the art for NLP applications like question answering or semantic search?"
  A: "Sebastian Ruder maintains a [list on github](https://github.com/sebastianruder/NLP-progress) of links to all the latest papers and datasets for NLP"
-
  Q: "Where should I start if I want to start getting up to speed on NLP."
  A: "_Natural Lanugage Processing in Action_ is an excellent book if you have the budget for it. Otherwise you can install the `nlpia` package at github.com/totalgood/nlpia and try the examples in each chapter on the repo. And here is another [great list of NLP resources]( https://github.com/sebastianruder/NLP-progress)"
-
  Q_teacher: what was the target variable in your past projects
  A: HIV positive woman screening, predict accepting or not cervical cancer screening test.
-
  Q_teacher: what featuer variables did you use to predict the target variable in that project
  A: income level, type of community/society (military, civil), age (16-48), height and weight, clinical screen
-
  Q_teacher: how did you determine causality
  A: individual linear regression and ANOVA
-
  Q_teacher: have you programmed before in visual basic or other programming languages like R, or C, or matlab?
  A: I created a data entry form using visual basic for my research project that helped improve the data entry error rate
-
  Q_teacher: how well did your interventions work and match your model predictions
  A: I didn't get a chance to see as I left the country after the research project (by Warren Buffet foundation) concluded
-
  Q_teacher: Shifting gears to python, what are the various data types in python and which one would you use for a table?
  A: int, float, list, tuple, dictionary. I'd use a dictionary for a table of numbers.
-
  Q_teacher: so if the keys were the column names in your dictionary for a table of numbers, what would the values be?
  A: a float? (no, it would probably need to be a list or numpy array of floats since), lists, arrays, tuples and dictionaries are called collections because they can contain multiple objects
-
  Q_student: I will not have too much trouble with the technical aspects. I hope you can help me with motivation and focus and understanding of the US culture since I am new to the country.
  A: The Springboard counselor and I will certainly keep you on track and help you make continuous progress. Also, I will help you chose among the many technical approaches by recommending the "best practice" for a professional data scientist in the US.
-
  Q: I broke up my number of weeks to complete the project into buckets 0-10, 11-20, ... etc. I created a categorical variable for this with an integer 0-9. How can I use that in my model as a feature?
  A: That's not a categorical variable, it's ordinal because it represents a quantity where the order of the categories have meaning, so you can pass it into your model just like any other.
-
  Q: What other models can I try with this data that I'm using to predict completion of our course?
  A: Random Forest may do a little better than logistic regression, but you may be running up against the limit of predictability of human behavior. A better approach would be to try to incorporate additional demographic data about students to create categorical variables that can be used as 1-hot encoded features to input into your model.
-
  Q_teacher: You're in an interview at Siemens, and they ask you to whiteboard a problem. "I have the crime dataset for Chicago and I'd like to use it to decide where to put bike racks for our rental U-bike product."
  A_student: You should estimate the crime rate at various locations throughout Chicago and put bike racks in low crime areas.
-
  Q_teacher: No. Don't ever reccommend a solution. Ask clarifying questions about the data and the problem before recommending a solution.
  A_student: OK, So what is in the crime dataset? Is it geocoded with latitude and longitude information? Does it have crime type categories? Does it have a category for bike theft? How many crime reports are there? Are there natural language descriptions? Over what time period do you have the reports? Is it live data that is generated every day or static?
-
  Q_teacher: What questions can you ask about the target variable and objective of the project?
  A_student: What are you trying to optimize? Sales? Theft? Minimize cost? Predict bike thefts?
  Q: Will discretizing my continuous variable to create an ordinal feature help reduce the noise and error in the output target variable predictions?
  A: No, it will not reduce the noise in this feature. Discretizing a continuous variable always reduces information. You should use the raw number in your model whenever possible. The best noise filter is the model itself. Consider a Random Forest or Decision Tree model, they will both discretize all your data with optimized thresholds betwee ordinal classes. And a linear or logistic regression, will average this noise over all the examples in your training set. So discretizing a feature is never a good feature engineering transformation.
-
  Q: How can I extract the last layer of an InceptionNet v3 for an video action classification model?
  A: The terms you want to DuckDuckGo is "transfer learning keras feature extraction". That should bring up excellent blog posts like this. The idea is to set the `include_top=False` when you load or instantiate the Keras model. That then allows you to use the `.predict()` method to output a tensor with all the features that were used by the network to predict the class variable it was trying to predict, like classifying images or doing object detection. You need to reshape that tensor into a vector for you new classifier and then train your model using that as your input feature vector instead of the array of frames (images) from the labeled videos in your training set. OpenCV may also have the ability to load Keras and TF models and extract any layer you like during activation. https://www.learnopencv.com/keras-tutorial-transfer-learning-using-pre-trained-models/
-
  Q: Should I take the natural language processing course or the deep learning course?
  A: In NLP you will learn deep learning concepts like CNNs and LSTMs and you will also learn the specific and complicated tasks of tokening strings and creating word embeddings. These more specialized skills will not likely be covered in the deep learning course. But the deep learning course will have more of the specialized skills for manipulating and processing images (computer vision). Much of your NLP skills will apply to deep learning computer vision problems, but if you are really interested in computer vision the DL class would be a better course.
-
  Q: Should I transform large dynamic range values, like word frequencies, with the log function so that they display more clearly in a plot?
  A: No, you should instead scale the axis with `plot(xlog=True)` or `ylog=True` so that the axis labels reflect the true counts in the data.
-
  Q: What is wrong with using net revenue as my target variable for predicting movie success.
  A: The relationship between budget and revenue is usually a nonlinear multiplicative rather than additive. So it would be more informative to train a model to predict the **ratio** of revenue to budget rather than the revenue minus the budget.
-
  Q: What is a good order to learn NLP concepts.
  A: The order of the chapters in my book, Natural Language Processing in Action, progresses in a logical incremental way building up from tokenization, to TFIDF
-
  Q: What's a good way to learn about the theory behind neural nets?
  A: Find a textbook from a university course.
-
  Q: How many samples do you need to get a **good** estimate of mean or standard deviation.
  A1: What do you think? What do you think the error in your mean and standard deviation is proportional to?
  A2: The error in your standard deviation and mean is proportional to `1/sqrt(N)`. MOst academics will accept a sample size of 30 for calculating a p-value and standard deviation and other critical statistics. But a sample size of 10 is usually sufficient for most practical real world problems. That will usually give you less than 10% error on your statistics.
-
  Q: I have a model that predicts the adjusted finish time of grayhounds, horses, or people. How can I incoporate information about how long ago it was that that runner raced, because I'm using their previous 5 adjusted finish times as features in my model.
  A: There's the easy way and the hard way and the super complicated way. The simple way is to compute the average adjusted finish time for each racer over the past N **days** rather than the previous N **races**. The hard way is to add a sample_weights array during training of your model that accounts for how long ago the last race was for each runner. And the most complicated way is to build a separate model that predicts adjusted finish time using **only** the adjusted finish times from the previous 5 races, with sample_weights proprotional to the time since those 5 races. Then you can use that prediction as a new feature to replace the previous 5 adjusted finish times as features in your model.
-
  Q: This dataset of rental bike inspection reports for New York city looks interesting buy I'm not sure how to use it as a time series or geo spatial data.
  A: "First, examine each column of data as if you don't know what the column heading means. How can you turn it into a number? How many numbers/features can you extract from each column? What kind of variable are those numerical features: categorical, ordinal, or continuous. What is the dimensionality of the features generated by each raw column of data? What feature engineering can you do on them to reduce dimensions or cluster them to make them more useful? And finally which column has a target variable (like the inspection report) that has information in it you would like to now before it is available in the dataset? What is the timeline for recording the columns in the dataset. Which values are known before the target variable? Would predicting the target ahead of time have business or scientific value?"
-
  Q: Should I try to translate a Caffe model to Keras like [this 2-stream model](github.com/wushidonguc/two-stream) to see how much optical flow helps with action prediction.
  A: That's a great research project or "future work" idea, but for this project all you need is a hyper parameter table listing all the parameters you tried and the performance as columns. The parameter columns can be things like image_shape/size, seq_length, batch_size, num_neurons layer 1, num_neurons layer 2, etc. The performance columns can be accuracy on your target for the training set and the test set or validation set.
-
  Q: What plots should I include in my final report.
  A: The money plot is few scatter plots (pair plots) of the model performance (accuracy) vs the most interesting hyperparameters. So the horizontal axis should be a parameter like batch size or num total neurons or num trainable parameters. The vertical axis should be test set and/or training set accuracy.
-
  Q: How do I do inferential statistics for Spam classification problem (dataset of spam and ham emails or messages).
  A: How did you do it for the discriminatory job interview callback statistics? How did you do it for the human body temperature dataset? You selecte a column of numerical values (categorical, ordinal or continuous feature values) and propose the hypothesis that there is a statistically significant positive (or negative) affect on the target variable. The target can be categorical (like job interview callback or not) or continuous (like body temperature). How could you do that for a dataset where your features are the frequencies (counts) of individual words or tokens and the target is the categorical (class) variable \"is_spam\"? You just pick a word likely to be predictive of spam and calculate the mean in the two classes and run a T-test or P-value test to see if you can discard the null hypothesis. Alternatively you can propose the hypothesis on all the words in your vocabulary in order and then sort them by their p-values to find the most significant predictors of spam. Or you can do dimension reduction (PCA or LSA) or clustering (KNN or LDiA) to generate aggregate classes for multiple words or topics and do your inferential statistics on those words.
-
  Q: "Do you know of any employers in the bay area? I'm interested in the less well-known ones."
  A: "Subscribe to the Owler, SiliconValley.com, silicontap.com, and siliconflorist.com and other newsletters/blogs that regularly report on startups receiving funding. In portland there are [slack](pdxstartups.com), Reddit, and IRC forums where startup news is shared. Meetups and \"Million Cups\" pitches are another great way to get plugged into your startup scene. Whenever you hear of an interesting startup receiving funding or hiring a CTO, visit their website to see if they have job openings in your area."
-
  Q_student: "What is LSA?"
  Q_teacher: "What does the LSA acronym stand for? What do each of those words mean to you? What do you think LSA means when you combine those words?"
  A_student: "Latent Semantic Analysis. Latent means hidden? I'm not sure what semantic means. Analysis means breaking something down into parts or computing numbers that describe the parts of something."
  A_teacher: "You're right. The word \"semantic\" means \"meaning\". So semantic analysis is breaking down a natural language document into its pieces of meaning. These are sometimes called \"topics\" or \"subjects\" when analyzing the meaning of natural language documents. So LSA is an unsupervised machine learning algorithm that creates a vector of topics or meaning for each of a large collection of natural language documents. The \"latent\" word is just an expression of researchers facination with the algorithms seeming ability to discover meaning in documents that was hidden from both humans and machines until LSA was invented."
-
  Q_student: "What is PCA and how is it related to LSA?"
  Q_teacher: "What does the PCA acronym stand for? What do each of those words mean to you? What do you think PCA means when you combine those words?"
  A_student: "Principal Component Analysis. \"Principal\" means \"most important\". \"Component\" means \"piece or part of something, often a dimension in a vector\". \"Analysis\" means \"breaking down something into its parts or pieces, usually expressed as numbers for each of the pieces within a vector\"."
  A_teacher: "You're right, and PCA is an algorithm that creates low dimensional dense vectors that express the topics or meaning within high dimensional vectors or tensors like images (light intensity arrays or tensors for each pixel), time series (prices or volume of transactions in vectors), word frequency vectors (bag-of-word vectors), DNA gene frequency vectors, any discrete fixed-length sequences or vectors, song audio sequences (audio amplitude vectors), etc.  When PCA is applied to natural language bag of words vectors (which have thousands of dimensions) it is called LSA (Latent Semantic Analysis) because the vectors PCA produces are semantic vectors that represent the meaning of each document in a lower-dimensional vector."
-
  Q: "Do you know of any data scientist employers in LA?"
  A: "For finance (quant) jobs check out all the banks in Santa Monica. To find lesser well-known companies, subscribe to the Owler and other newsletters/blogs that regularly report on startups receiving funding. In portland there are [slack](pdxstartups.com), Reddit, and IRC forums where startup news is shared. Meetups and \"Million Cups\" pitches are another great way to get plugged into the startup scene in any new area. Whenever you hear of an interesting startup receiving funding or hiring a CTO, visit their website to see if they have job openings in your area."
-
  Q: "Isn't machine learning for optical flow a trivial problem."
  A: "No, it's an active area of research. And if you can come up with some preprocessing, like FFT or optical processing, like microlens arrays, that reduce the latency and computation required it would be extremely valuable in self-driving car and robotics applications."
-
  Q: "The fringe lines produced by a weighted aveage of 2D spatial FFTs should be perpendicular to the optical flow. And L shaped features, like the windows in this self-driving car video should produce X's in the averaged 2D spatial FFT image. So shouldn't I figure out how to combine the two sides of the X to compute the optical flow for that subwindow?"
  A: "The averaged 2D spatial FFT fringe lines look like they are parallel to optical flow to me because they point out radially from the center of the FOV for a camera facing forward on a self-driving car application."
-
  Q: "I have data for water consumption over 15 minute intervals. I also have flags like, 'is_sports_event', 'is_holiday', 'is_weekend', 'is_rush_hour', 'is_business_hours' for each 15 minute interval. And there's a previous 3 hours of precipitation value as well. Some values are nan, particularly in the precipitation and water consumption columns. How can I build a time series model to predict the next day's water consumption?"
  A: "
  You need to use `DataFrame.ffill()` to fill `NaN` values rather than filling with the mean. Any data preprocessing that brings in future data to the past will help your model cheat. Next, you need to aggregate all the columns for each day using the DataFrame.resample(). Then you can time shift the precipitation water consumption data like this. Here's some fake data with column 'c' as your consumption data. You have to be careful to avoid Pandas' indexing magic which will try to prevent you from shifting your data:
  ```python
  >>> pd.np.random.seed(hash('Peter Watts') % 2**32)
  >>> N = 7
  >>> columns = list('abc')
  >>> df = pd.DataFrame(
          pd.np.random.rand(N, len(columns)),
          columns=columns,
          index=pd.date_range('2020-01-01', periods=N, freq='1d'))
  >>> df = (df * 8).astype(int)
  >>> df
  a  b  c
  2020-01-01  4  4  5
  2020-01-02  4  5  5
  2020-01-03  6  0  3
  2020-01-04  5  3  6
  2020-01-05  6  7  1
  2020-01-06  0  7  5
  2020-01-07  3  6  2
  >>> df['c_tomorrow'] = pd.Series(df['c'].iloc[1:].values, index=df.index[:-1])
  >>> df
              a  b  c   c_tomorrow
  2020-01-01  4  4  5          5.0
  2020-01-02  4  5  5          3.0
  2020-01-03  6  0  3          6.0
  2020-01-04  5  3  6          1.0
  2020-01-05  6  7  1          5.0
  2020-01-06  0  7  5          2.0
  2020-01-07  3  6  2          NaN
  Now you can perform any old machine learning model `.fit()` on the 'abc' columns as your indicator features and 'c_tomorrow' as your target output variable.
  ```
  "
-
  Q: "In Medicare regulations, Fraud Waste and Abuse (FWA) are 3 different categories of laws and regulations. Which one requires the **intent** to obtain payment as well as the knowledge that the actions are wrong or illegal?"
  A: "A Medicare FWA fraud conviction requires that the violator intended to obtain payment and knew it was wrong."
-
  Q: "In Medicare FWA regulations, which is not potentially a penalty for a violation: Civil monetary penalties, Deportation, or Exclusion from participation in Federal healthcare programs?"
  A: "Deportation is not a potential penalty for violating Medicare FWA regulations. "
-
  Q: "In Medicare FWA regulations what is an FDR?"
  A: "In Medicare FWA regulations an FDR is a Firs-Tier (call center, hospital), Downstream (provider), or Related Entity (contractor, researcher). All such organizations must have policies an procedures that address FWA."
-
  Q: "In Medicare FWA regulations what is an SIU"
  A: "In Medicare FWA regulations an SIU is a Special Investigations Unit within a sponsor organization, such as a Firs-Tier (call center, hospital) participant in the Medicare program. An SIU will investigate any reports of FWA, including anonymous reports."
-
  Q: "Where can I report Medicare FWA violations?"
  A: "1-800-HHS-TIPS (1-800-447-8477) or HHSTips@olg.hhs.gov or forms.oig.hhs.gov/hotlineoperation/index.aspx"
-
  Q: "In Medicare FWA regulations, what is OIG?"
  A: "In the Medicare FWA context, OIG is the Office of Inspector General responsible for investigating and prosecuting FWA."
-
  Q: "What are some common indicators of Medicare FWA in a medical dataset for beneficiaries or patients?"
  A: "Altered or forged prescriptions, medical records, or laboratory test results; Beneficiaries medical history doesn't support the requested servics; identical prescriptions for the same beneficiary, often from different doctors; beneficiary identity mismatch (identity theft); innappropriate prescriptions based on the beneficiary's other prescriptions (dangerous drug interactions or contrary indications)."
-
  Q: "What are some common indicators of Medicare FWA in a medical dataset for providers or doctors?"
  A: "Medically unnecessary or harmful prescriptions; services not provided but billed; provider writes prescriptions for controlled substances or a broad variety of drugs often; medically unnecessary services provided to beneficiary; unneccessarily high quantities of prescriptions/procedures; invalid or missing Ntional Provider Identifier on prescriptions; unsupported diagnoses based on the medical record."
-
  Q: "What are some common indicators of Medicare FWA in a medical dataset that includes pharmacy data?"
  A: "Drugs meant for nursing homes, hospices, etc sent elsewhere; dispensed drugs are expired, fake, diluted, or illegal, generic drugs when prescription requires branded drug; PBMs billed for unfilled or never picked up prescriptions; additional dispensing fees for split prescriptions; altered prescriptions like changed quantities or 'Dispense As Written' notation added to prescriptions."
-
  Q: "What are some common indicators of Medicare FWA in a medical dataset that includes wholesaler data?"
  A: "Wholesaler delivering/distributing fake, diluted, expired, or illegal drugs; wholesaler diverting drugs intended for nursing homes, hospices, and AIDS clinics, marking up the prices and sending them to other small wholesalers or pharmacies."
-
  Q: "What are some common indicators of Medicare FWA in a medical dataset that includes manufacturer data?"
  A: "Manufacturer promotes off-label drug usage; wholesaler providing samples to entities (providers) that are also billing Medicare for those same drugs."
-
  Q: "To avoid Medicare FWA what should you do if a regular beneficiary suddenly has an increase in their quantity or dosage for a prescription or controled substance?"
  A: "Contact the provider to verify the quantity."
-
  Q: "Within the medicare and medical field, what is CMS?"
  A: "Centers for Medicare & Medicaid Services"
-
  Q: "Where can I find medical or healthcare datasets?"
  A: "Search for papers on https://scholar.google.com/ that interest you and try to regcreate the results in those papers using any data that they link to or referecnct. Browse or search https://github.com/topics/datasets for datasets that interest you. Google's [dataset search tool](https://toolbox.google.com/datasetsearch/) can help you find obscure datasets. Put a search into Duck Duck go that includes words like 'machine learning data science download zip gz csv data training set test set validation'"
-
  Q: "What is the computational complexity on an unsupervised algorithm to recommend commercial building retrofits to minimize energy consumption? Your goal is to find similar buildings and put them in clusters."
  A: "O(N^2)"
-
  Q: "What is PCA when you apply it to bag-of-word vectors?"
  A: "LSA, Latent Semantic Analysis. PCA.params_ will contain a matrix of word to topic weights that can be multiplied by any high dimensional bag-of-words vector (with that same vocabulary) to produce a low-dimensional topic vector."
-
  Q: "What goes in the presentation slides for the capstone?"
  A: "A summary of all the interesting things you discovered in your report. Visualizations and statistics that you think would be interesting to your audience."
-
  Q: "Because I have mostly categorical variables my correlation matrix with my target variable does not have many continuous correlation values."
  A: "You need to use pd.get_dummies to create one-hot encodings of your categorical variables and you need to encode your ordinal variables and consecutive integers so that `df.corr()` will display all the correlations that you are interested in."
-
  Q_teacher: "An employer asks you to create a recommendation engine for a recruiting firm. They have a database of resumes and job descriptions with some structured metadata for each (job title, location, name, email). They have information about past candidates and the jobs they applied for, interviewed for, and were hired for. What approach would you recommend?"
  A_student: "There are two general approaches for recommendation engines, collaborative filtering and content-based filtering. A content-based approach would use NLP to compare all pairings of job descriptions and resumes to look for good matches. A collaborative filtering approach would cluster candidates and job descriptions and recommend jobs to candidates that were a good match for others in their cluster. "







-
  q: "How to use chi squared test in feature selection?"
  a: 'Personally I find it hard to know ahead of time what the right threshold for chi**2 test is. Instead I use automatic dimension reduction techniques like PCA, regularization, and hyperparameter tuning (grid search). These require less "judgement" on my part. I let the data and algorithm make the decision for me.'
-
  q: "What's the benefits of using log loss vs accuracy? And usually where log loss is used? (do we just treat for example 5 % quantile as outliers?)"
  a: "There are numerous cost/loss functions you can use, or you can make up your own. Almost all of them will give you the same answer to your optimization/fitting problem. But some will be better or worse at telling you how good your model is at predicting the things you care about. Sometimes you care about the standard error, sometimes you care about the log of the computer standard error, sometimes accuracy is more important to the success of your project, sometimes precision, sometimes recall, sometimes F1 or AUC. What do you think the agile approach would be to answering your question? Is there a way to use my answer to 1 to come up with an approach to question 2? Is there a way to ask your \"boss\" or customer which one matters most to them? What is the measure of units that most businesses measure success or lose (cost ) with? What are the best \"units\" for business decisions?"
-
  q: "How to detect outliers in a data set and how to handle them?"
  a: "Like other pipeline design challenges, I do outlier filtering in an agile way. Can you guess what that is? Spoiler alert... Outliers can be discarded automatically during hyperparameter tuning, or even within the model itself (random forest or xg-boost)"
-
  q: "Can I use .corr() or pierson correlation on a binary or categorical variable?"
  a: "Correlation is a mathematical statistic. It can be calculated on any numerical value where that numerical value contains information about the real world (integer ordinal, binary, float, or bool, but not a categorical variable until it has been one-hot encoded). What do we do with all categorical features to create numerical values that work in a machine learning pipeline? Hint: correlation is equivalent to linear regression."
-
  q: "Can I use .corr() or pierson correlation on an integer feature?"
  a: "think deeply about how an integer is different from a float. Is 2.0 Twitter likes the same thing mathematically as 2 Twitter likes ?  30.0 vs 30 years old ? What about 5.0 stars vs 5 stars? What about toaster oven low med high vs a numerical toaster knob with 0, 1, 2 or 0.0 1.0 2.0 ? What about word frequency of 0/1 vs TF-IDF of 0.0/1.0 vs Boolean word presence value of False/True ?"
-
  q: "Can I use .corr() or pierson correlation on an integer or binary feature?"
  a: "Each situation is different. You have to think about the \"physics\" of something to know what what the best numerical representation is. We can go through some examples to help you learn how to think about numerical value types like binary, int, float, boolean, categorical, ordinal, and continuous things in the real world."
-
  Q: "How can I get more confidence in how the time series predictions will work on real data in the future? "
  A: "Implement an expanding window validation and model training pipeline that trains 100s of models using the last day as the validation example."
-
  Q: "how to check the correlation b/t two variables between an integer and a continuous value? For example how many people likes a comment on twitter vs the age of user?"
  A: "Ordinals (integers) can be treated just like a continuous value throught a machine learning pipeline, including the correlation coefficient calculations."
-
  Q: "how to check the correlation b/t two variables between an integer and a boolean value? For example whether a user is an engineer or not. "
  A: "Booleans (True/False) can be treated just like a continuous value if they are convert to integer 0/1 values or floating point 0.0/1.0 values. They'll work throughout a machine learning pipeline, including the correlation coefficient calculations."
-
  Q: "Is pearson correlation coefficient work for both for booleans, ordinals (integers)?"
  A: "Yes. Booleans and ordinals are cast to floats (continuous values) and should work fine in any correlation coefficient or any other statistics calculation."
-
  Q: "What is a Markhov Chain model?"
  A: "A model of state transitions (sequences like time series) where the next state depends only on the current state and not any of the previous states. Example: predictive text that only looks at the last token you typed (or the \"<start>\" token) to predict the next token."
-
  Q: "What is a Markhov Chain Monte Carlo model?"
  A: "A Markov Chain that estimates the distribution for the next state by sampling historical data or a simulation. Example: predictive text that only looks at the last token you typed (or the \"<start>\" token) to predict the next token and estimates the distribution of possible next tokens based on historical data for all the texts you (or people like you) have typed in the past."
-
  Q: "What is a static model or distribution?"
  A: "A probability distribution or model whose underlying statistics or behavior do not change over time."
-
  Q: "What is a deterministic model?"
  A: "A model or algorithm that has a closed form solution (can be solved by evaluating a sequence of mathematical expressions) without relying on random sampling or other stochastic (semi-random) algorithms. A stochastic model or algorithm should converge to the same solution to a given problem (for the same dataset) regardless of when or where it is trained."
-
  Q: "What is a nondeterministic model?"
  A: "A model or algorithm that has no closed form solution and relies on stochastic or random sampling techniques to achieve a solution.$"
-
  Q: "Give examples of deterministic and nondeterministic models."
  A: "Stochastic: KNN, K-Means, Stochastic Gradient Descent? Neural Networks with random dropout or random initialization of weights, Random Forest, Decision Tree. Deterministic: Linear Regression, PCA, SVD, Logistic Regression, Polynomial Regression, Stochastic Gradient Descent?"
-
  Q: "What are the characteristics you look for in a Data Science or Machine Learning model?"
  A: "Training time/complexity per data sample and feature (dimensionality). Inference time/complexity. Data efficiency. Number of parameters (degrees of freedom). Linearity/nonlinearity of the solution. Nonlinearity/complexity of the generated model. Classifier or regressor. Supervised or unsupervised. Stochastic or deterministic."
-
  Q: "What is a leverage plot and what should I look for?"
  A: "I don't know."
-
  Q: "What is a residual plot and what should I look for?"
  A: "you want to look for any patterns or anomalies in the data. If the residuals are not normally and uniformly distrubuted, that's an opportunity for you to do additional cleaning, like removing outliers from your training set (but not your test set), filling in missing or incorrect values. If there's some clear curvature to the data then that's an opportunity to do some more feature engineering, like squaring or rooting or taking the log of all your features to create new features."
-
  Q: "What is a quantile plot and what should I look for?"
  A: "I'm not sure, but it seems to show the normal statistics (z-score) variation across various values of your target variable. It seems to plot the standard deviation and mean of your predictions in a rolling window across a sorted list of your target variable (like home price). So it reveals oportunities for additional feature engineering on the \"tails\" or edges of the plot, where you may have fewer examples and the model may need to include nonlinearities to deal with _edge cases_."
-
  Q: "What is False Discovery Rate?"
  A: "It's a way to adjust your P-Value or T-test or Z-test or X^2 test to account for the number of experiments you've run, then number of times you've compared your P-value to your 5% threshold (typically). If you only do one experiment you can be 95% confident that your results are going to be repeatable by others, if your P-value is below 5%. But if you do 2 experiments, an only one passes the p-value threshold test, then you can only be 91% confident. So to keep your confidence high, you need to tighten your P-value threshold based on the number of tests you've performed on a given dataset."
-
  Q: "What is the Kernel Trick?"
  A: "It's when an ML algorithm like SVM transforms the feature vectors using a linear or nonlinear kernel function. For example there may be a  linear rotation and scaling matrix that unwinds and flattens a spiral classification dataset (there's an example in playground.tensorflow.org)."
-
  Q: "And what does it mean when someone says that an algorithm like SVM doesn't actually transform the data with the kernel?"
  A: "The kernel can be selected and optimized within the cost function calculation of a SVM or SVC, so it's applied to the loss rather than to the higher dimensional feature vectors to save computational effort (I think)."
-
  Q: "Where can I find some good DS FAQs or anwers to common questions?"
  A: "In addition to this knowledgequest chatbot, you can try places like Springboard.com's community Kahn Academy and Kaggle.com. For deeper questions you should do some searches on scholar.google.com."
-
  Q: "Recruiter contacted me about jobs in Houston for DS, Sr DS and Data Analytics. The data analytics position looks like the work I already do, but it's a bit advanced. What should I do?"
  A: "Apply for all of them, and read the job descriptions with an eye for the experiences you've had on Springboard and at work with each of the tools, languages, challenges in those job descriptions. Take notes on how you used the tool, or solved a problem at work, whenever you have an interesting problem or tool or learn something from the other DSs or developers."
-
  Q: "The job descripotion mentions Spark, what should I do?"
  A: "Make sure you know what Spark is and think about all the places you've used it on Springboard or at work. What problems is it good at? Paralellizable machine learning pipelines that need to scale to more data than fits into RAM. The employer won't expect you to be able to code up a Spark pipeline during a code interview. But they will ask you to analyze a problem and describe how Spark might be helpful. "
-
  Q: "The job descripotion mentions ETL/ELT, what should I do?"
  A: "What does ETL stand for? \"Extract Transform Load.\" What does ELT mean? \"Extract Load Transform.\" Are they different in some way? Which acronym do you prefer? Which one makes more sense based on what your experience with ETL? Why? I prefer \"ELT\" even though it isn't as common as ETL. ELT is the order that I typically have to do things during the data cleaning and visualizaiton process. But in a large database, you may have to do a lot of the transformations incrementally on only portions of the DB or individual records, loading them one at a time and transforming them. So both acronyms make sense. Make sure you know what the hiring manager means by the terms Extract, Transform and Load terms and think about all the places you've encountered similar problems and tools at work or on Springboard. The hiring manager and team may ask about any experience you have with ETL and any insights you have about it, like how do you deal with missing values? How can you automate some of the most labor intensive parts of ETL? What are the challenges of ETL? What are some good ETL tools?"
-
  Q: "Why does GridSearchCV take so long? What's wrong with my code?"
  A: "GridSearchCV trains 3 different models on the data, by default, because it uses KFolds cross-validation with K=3. In addition, your implementation of your `create_model()` function trains and validates a Keras neural net model every time you call it. So the GridSearchCV is retraining and revalidating your model 3 times, for a total of 6 trainings.  Never use the black-box functions that do the work of a data scientist, like hyperparameter optimization or cross-validation. Training and evaluating model performance usually requires a lot of computation and memory and time, and you want to minimize that time by inserting your DS intuition into the optimization process. If you do want to automate a grid search use `itertools.product()` to manually iterate through your hyperparameter combinations and evaluate your model performance."
-
  Q: "What's the right way to do hyperparameter optimization?"
  A: "Bayesean search rather than grid search, or human-intuition-driven manual search. If you don't have probability distributions in mind that you want to test, just assume that the prior is a continuous distribution across a range of values or a list of discrete values. Check out the `hyperopt` package on github and pypi. If you can't figure out how to implement bayesean search, then just do a random sort of your grid search product(): `sorted(product(param1_list, param2_list, ...))`. And if you really want to speed things up, Start with random search for the values that minimize the training and inference time for your models. For parameters that increase the training time significantly, only try them rarely, and only with one parameter at a time."
-
  Q: "What are replicants in the context of bootstrapping?"
  A: "I don't know but I imagine they are the duplicates that occur due to random sampling with replacement."
-
  Q: "What kinds of problems are there in the inferential statistic exercises?"
  A: "Problems that indicate your understanding of Z-score, T-tests and other statistical measures of how likely it is that a statistic, like mean or sum, is significantly different in one group (the treatment vs the control)."
-
  Q: "What is the difference between Data Wrangling, Data Storytelling, and EDA (Exploratory Data Analysis)?"
  A: "What does data wrangling mean to you? Loading, cleaning, and organizing your data into a form suitable for visualization and modeling. Generally that means consolidating your data into a single table. What does data story telling mean to you? It's building a story about the real world that the data is telling you. You need to have done EDA and data wrangnling first so you have plots and statistics about your data that can help you come up with those ELI5 descriptions of what's going on in the real world to cause that data to appear in your database. What are people clicking on a particular button? What are men generally taller and denser, on average, than women? Those are examples of questions that data story telling might answer. What does EDA mean to you? Exploratory Data Analysis is the plotting and statistics calculations you do on your data to get a better understanding of what's going on in the real world to create those data entries. You want to look for patterns and correlations between features to see if they make sense or contradict your intuitions. These patterns will suggest models and statistical tests that you can use to build a Data Science pipeline and a data story."
-
  Q: "Are these plots useful or should I remove them?"
  A: "What are you trying to convey to the reader? What do you want them to get out of them? What do they tell you? Is that piece of information obvious at first glance? You reader will likely only look at the plot for a few seconds. In a bar plot, the bars next to each other should always have a relationship to each other that is interesting, since those will be compared to each other. If you have more than 10 bars on a single plot, they will likely hide the important pairs that you want your reader to look at, so cluster or group them more agressively. And never ever use raw counts on a bar plot or histogram, always us percent or normalized count so that your plot shows probability and allows your reader to compare values from different histograms with different total counts (like the cohorts in your income survey data)."
-
  Q: "The conclusion to my capstone starts with \"The above plots show that feature importance are driving model performance.\" What do you think?"
  A: "I'm not sure I understand what you are trying to say with that. Are you saying that the features with high feature importances values from the random forest model statistics are important to the model's decision tree branches? That's a circular statement that doesn't say anything new. Talk about the real world affects of your features on your target, income level. Talk about the fact that relationship status questions can have both a postiive and a negative impact on income. \"married with wife\" relationship status is highly positively correlated with income, whereas \"married with children\" relationship status is highly negatively correlated with income. And talk about how education level is not highly correlated with income, above a high school diploma. Those are interesting, meaningful, specific conclusions you can share with your reader."
-
  Q: "Which penalty function should I use for logistic regression, l1 or l2. My dataset has the heights and weights of hospital patients and the target variable is gender."
  A: "What do you think is the right way to chose between two different possible settings or values for a hyperparameter, like the regularization penalty metric for logistic regression? Try it, right? Whichever one improves your performance on the test set is the **right** one, right? And for this particular model your degrees of freedom is only 2 (`len(np.array(model.coef_).flatten())`) and the number of examples is presumably much larger than 2, probably 100s or even 1000s. So theres almost no change your 2 DOF logistic regression can overfit. So it doesn't matter what C value (inverse regularization strength) you chose and which penalty metric you choose."
-
  Q: "What is the difference between l1 and l2 penalty or distance? How does each one affect the regularization penalty in a logistic regression."
  A: "l2 penalty is crows flies distance. L1 penalty is manhattan distance. The 1 and 2 come from the exponent on each feature/dimension value that is applied before summing up the individual dimension distances to compute the distance. So a manhattan (l1) distance metric is usually better because a drop in an individual dimension coefficent in your model will have a greater overall impact on the penalty than for l2. Consider the diagonal of a 1x1x1 cube for a 3 DOF linear regression model with slopes of 1, 1, and 1, vs 1, 0, and 0. The first model would have an l2 penalty of sqrt(3)=1.7 and the second, much simpler model, would have an l2 penalty of 1. That's a drop of 42%. For manhattan (l1) distance the penalty would drop from 3 to 1, gives a drop of 66%."
-
  Q: Which penalty function should I use for logistic regression, l1 or l2. My dataset has the heights and weights of hospital patients and the target variable is gender.
  A: What do you think is the right way to chose between two different possible settings or values for a hyperparameter, like the regularization penalty metric for logistic regression? Try it, right? Whichever one improves your performance on the test set is the **right** one, right? And for this particular model your degrees of freedom is only 2 (`len(np.array(model.coef_).flatten())`) and the number of examples is presumably much larger than 2, probably 100s or even 1000s. So theres almost no change your 2 DOF logistic regression can overfit. So it doesn't matter what C value (inverse regularization strength) you chose and which penalty metric you choose.
-
  Q: What is the difference between l1 and l2 penalty or distance? How does each one affect the regularization penalty in a logistic regression.
  A: How do you chose any hyperparameter? What does l1 penalty mean? What does l2 penalty/distance mean?
-
  Q_teacher: If you're given a data set with 10 columns 1M rows of numbers and the 10th column is the target you are trying to predict. What do you do to get started?
  A_student: df.describe(), df.info(), assume any low cardinality integers are categorical (get_dummies). Scatter plot of each feature vs the target. Linear regression, if no overfitting, random forest, if overfitting add regularization or feature reduction.
-
  Q: How does one decide what to do next with a 2-layer 128-cell LSTM network that is getting 95% training set accuracy and 91% test set accuracy for sentiment analysis? Does it have high variance or high bias?
  A: I don't like using those terms what's a better way to describe how your model is doing? (Overfitting vs underfitting)  You model is overfitting so you need to increase the random dropout and reduce the number of layers or the number of neurons in each layer. In general the only way to decide what to do next is to try it. And record your answers in a hyper-parameter tuning log or table. And record each new model as a separate python file or notebook so your can reproduce the results of previous models. Especially when it takes a long time to train.
-
  Q: should I get more data to reduce overfitting?
  A: getting more data is usually the harder thing to do so you should do it last. Instead try to make your model smarter by increasing regularization and reducing the number of features or doing feature reduction like PCA.
-
  Q: Is scaling and normalization useful for a Linear Regression? What about SVM?
  A: No, in general vastly different scales do not affect the performance of most ML models like linear regression. However a Support Vector Classifier and Support Vector Regressor do tend to benefit from normalized scaling of the data (for example `sklearn.preprocessing.MinMaxScaler()`). This is because an SVM optimizes the margin between target values/class, and normalizing your feature values ensures that the margin is balanced across the features. For example, the margin/separation between the class boundary for a feature ranging over +/- 100 would be 100x further away from the "true" class boundary than for a feature ranging between +/- 1, if feature normalization or scaling were not used.
-
  Q: My tensorflow+CUDA installation is not recognizing my GPU when I train a model. What should I do?
  A: What is your OS, CUDA, python, and tensorflow version. Mine works for CUDA 9.0 on Ubuntu 18.04 with python 3.6 and tensorflow 2.0. In addition, it may help to set the global environment variable `export CUDA_AVAILABLEGPUS="0"` if you have 1 GPU or `export CUDA_AVAILABLEGPUS="0,1"` if you have 2 GPUs.
-
  Q: Is the movie review dataset a good one?
  A: Yes.
-
  Q: "SQL is not applicable to what I want to do. Do I need to do these exercises?"
  A: "I don't use SQL much in my everyday work as a data scientist and machine learning engineer. However as a backend software engineer it will be a critical skill. And I've had many Data Science interviews where I asked or was asked SQL questions. Even people who only want to use Tableau to explore and visualize data often have to use SQL. One way around this is to use an ORM, like `Django` or `sqlalchemy`."
-
  Q: "Is SQL required for DS work?"
  A: "No, but it's required to get through many interviews."
-
  Q: "The inferential statistics exercises are hard."
  A: "Inferential statistics is all about using classical Data Science statistical approaches to create basic models to predict a target varialbe. And example of an inferential statistics model would be if you tried to predict gender based on only the height and weight of people. You could take the mean height and weight of the women in your training set, and likewise for the men, and then assume that whichever centroid your test examples are closest to can be used to assign them a predicted gender label. Obviously this model has flaws and a machine learning approach would involve creating additional features, like perhaps dividing the height by the weight and then finding the optimal threshold on this estimate of BMI (body mass index) to predict the gender (whichever threshold gives you the highest accuracy, precision, or recall, whatever you care about most). "
-
  Q: "How can a testing engineer position interview (asked about DS applied to test engineering) be parlayed into a Data Science position."
  A: "Once you have the job as a test engineer, you can show your colleagues how to predicting and diagnose issues with hardware or software (whatever you're testing) using machine learning."
-
  Q: "I'm in sales and marketing. How can I apply AI to my work?"
  A: "You can use ML (a form of AI) to predict customer behavior, like favorable or unfavorable response to a marketing campaign or the sales generated by a particular promotion. You could even build a chatbot or a generative model to perform that advertisement for you. Google is a leader in AI research and their focus is on advertising and marketing, mostly through the AI that is part of their search engine and ad-words program."
-
  Q: "What are some ways to learn about marketing, and marketing techniques that apply ML."
  A: "There are numerous blog posts and academic papers on applying ML to marketing and advertisement. Search for \"advertising marketing machine learning\" on DuckDuckGo.com, toolbox.google.com/datasetsearch, kaggle.com/datasets,  and scholar.google.com."
-
  Q_teacher: "What is a good way to plot data that has widely varying scale like the frequency or document frequency for the 50 most frequent words in my documents?"
  A_student: "I don't know, maybe zoom in on one part of the data? Oh I know, how about a log scale plot so that all the values have equal visibility."
-
  Q: "What values for min_df and max_df I should you use for the `sklearn` `TfidfVectorizer()` ?"
  A: "What does df mean? What does min_df do? What does max_df do? How does it help a model? What is the purpose. How does df vary across the words in your documents? How many documents would be affected if you increased the min_df by one? How much would your vocabulary size be reduced? How much would this help your model? What if you decreased max_df by 1 or by 1% (.01), how many documents would be affected? How much would this help?"
-
  Q: "Is it a good data science project to build a smart search tool for GEO and Environmental dataset like oil pipeline locations, whale and marine species migration data, farming data, etc."
  A: "No. Just gathering up the metadata about those datasets would be very difficult and time consuming. The best project ideas are not ideas that I generate on my own, they are ideas that are suggested by a dataset a table with numbers in it."
-
  Q: "What is a good dataset to practice joining datasets or tables?"
  A: "The conventional approach is to find an existing data science problems like predicting home prices and predicting crime in Chicago and then join them to create additional features for both problems. The crime data joined on the latitude and longitude or street address in both tables could be useful as a feature in predicting home prices. And the home prices and number of stories and other features of homes might be useful in predicting crime rates if joined with the crime table on the street address or geolocation of the crime reports."
-
  Q: "What about joining the Kaggle competition on predicting the dish name based on the ingredient lists in recipes, with another table from grocery stores about their stock of those ingredients. Is that a good Data Science project?"
  A: "Is it going to be easy to obtain a reqpresentative sampling of grocery store stock and prices from across the united states? It's not. So it is better to let the datasets suggest a project rather than letting you mind dream up hypothetical datasets that might be hard to obtain. For example can you think of a way to use the FDA.gov website which has CSV files of food items and their nutritional content? What if you joined it on the recipe dataset from Kaggle? What would your target variable be? (any of the nuttitional components available on the FDA databse, like sugar or fat or calories or vitamin C) Would it be supervised or unsupervised? (unsupervised) Use toolbox.google.com/datasetsearch and kaggle.com/datasets to help you come up with a dataset (or pair of datasets) and a problem statement."
-
  Q: "For hyperparameter tuning, feature selection in particular, if I add a new feature to my model, how do I know if it helped or hurt. "
  A: "If you see an improvement on your training set accuracy but your test set accuracy is degraded then the feature likely doesn't contain useful information for the model, so it can be skipped."
-
  Q: "For hyperparameter tuning, feature selection in particular, should I train a model based on a single feature at a time before training on combinations? "
  A: "It will be more efficient to simply accumulate more and more features by adding features one at a time and not training on them each independently. If you don't see an improvement in the training set or the test set accuracy, then you can likely skip it. Or you can keep it in and wait until your test set accuracy is degraded before your ignore features."
-
  Q: "For hyperparameter tuning, should I record the model coefficients using something like `model.save()` with every new combination of features or hyperparameters that I train it on?"
  A: "Not if the model is relatively fast to train/fit.  Because you've recorded your hyperparameters in a hyperparameter tuning log or table you can always recreate any of the models that worked well by refitting it using those hyperparameters and the training set you used the first time.  If the model is complicated with a lot of coefficients and takes a long time to train you can save a \"checkpoint\" whenever you see an improvement in your test set accuracy, so you don't have to try to recreate it later."
-
  Q: "What are the characteristics you look for in a Data Science or Machine Learning model?"
  A: "Training time/complexity per data sample and feature (dimensionality). Inference time/complexity. Data efficiency. Number of parameters (degrees of freedom). Linearity/nonlinearity of the solution. Nonlinearity/complexity of the generated model. Classifier or regressor. Supervised or unsupervised. Stochastic or deterministic."
-
  Q: "What makes a good dataset?"
  A: "A large number of labeled samples is important to make the problem straightforward. high dimensionality can make the problem harder. Examine the relationship between possible target variables and the other features to make sure the target is something that would be interesting to \"predict\" if you only had access to the features."
-
  Q: "Should I look for something I'm interested in, or something that would be a good project?"
  A: "Both. Start with the dataset, like at the google dataset search tool. Look for interesting datasets where the data is arranged in tables that you can easily import an manipulate (CSVs with numerical values). Think about the dimensionality of the features and the number of samples/examples/rows and whether you think the problem will be solvable. Examine the relationship between possible target variables and the other features to make sure the target is something that would be interesting to \"predict\" if you only had access to the features."
-
  Q: "How can I view the sparse matrix output of a `vectorizer.transform()` method, like sklearn.CountVectorizer().transform()?"
  A: "Yes, try `doc_vectors = pd.DataFrame(sparse_matrix, columns=vectorizer.get_feature_names())`."
-
  Q: "For my library usage survey data, how do I visualize the `.corr()` matrix in a way that makes sense?"
  A: "Set the diag() of your correlation matrix to 0 so that it doesn't swamp the plot. Also display the correlations as a single column at a time with `C=df.corr(); C['colname'].sort_values().round(2)`. Also, in your heatmap, use seaborn and set the axis labels to the column names with `ax.set_xaxislabels(C.columns)`."
-
  Q: I don't have a time series so that doesn't apply.
  A: Don't you have a survey submission datetime? That makes it a time series.
-
  Q: "What are some features you can extract from a datetime?"
  A: "Here are some binary feature examples: 'Is it a Holiday?', 'Is it summer?', 'Is it the weekend?', 'Is it Sunday?', 'Is it December?', 'Is it morning?', 'Is it lunch hour?'.  Can you think of some more binary features? What about some categorical features? What about some ordinal features?"
  A_student: "'Is it daytime?' and 'Is it nighttime?' or 'Is it a full moon?'. Some ordinal features might be the 'phase or brightness of the moon', or perhaps 'the angle of the sun to the horizon' or 'how close to noon it is') could be ordinal or continuous features."
-
  Q: "How do I know whether 12 LSTM cells is enough. I acheieved 91% sentiment classification accuracy with 128x2 LSTM cells and I achieved 90% accuracy with 12x1 cells."
  A: "The only way to find out is to try. Use Daniel C Bennet's intuition pump \"make mistakes.\" That's the best way to learn and develop an intuition."
-
  Q: "What does a hyperparameter tuning table look like?"
  A: "It has a column for each hyperparameter that you changed and a row for each model that you trained. You should also have a column for the test set accuracy or loss. This is what you care about the most. You can also have the training set accuracy and maybe the test set accuracy at epoch 10, or some other arbitrary point in your training."
-
  Q: "What kind of accuracy should I expect for sentiment analysis."
  A: "92% accuracy is probably the upper limit for a large clean dataset like the Yelp restaurant reviews dataset and a sophisticated LSTM model. On other datasets 80% or even 75% may be the best you can achieve. Sentiment is a very gross target variable that can contain a lot of noise. Above 90% accuracy, you are probably starting to fit to noise. A human cannot likely achieve this accuracy, so a machine that does is superhuman, despite not having common sense knowledge or empathy."
-
  Q: "What is the state of the art for NLP applications like question answering or semantic search?"
  A: "Sebastian Ruder maintains a [list on github](https://github.com/sebastianruder/NLP-progress) of links to all the latest papers and datasets for NLP"
-
  Q: "Where should I start if I want to start getting up to speed on NLP."
  A: "_Natural Lanugage Processing in Action_ is an excellent book if you have the budget for it. Otherwise you can install the `nlpia` package at github.com/totalgood/nlpia and try the examples in each chapter on the repo. And here is another [great list of NLP resources]( https://github.com/sebastianruder/NLP-progress)"
-
  Q_teacher: "What was the target variable in your past projects?"
  A: "HIV positive woman screening, predict accepting or not cervical cancer screening test."
-
  Q_teacher: "What featuer variables did you use to predict the target variable in that project?"
  A: "Income level, type of community/society (military, civil), age (16-48), height and weight, clinical screen."
-
  Q_teacher: "How did you determine causality."
  A: "Individual linear regression and ANOVA."
-
  Q_teacher: "Have you programmed before in visual basic or other programming languages like R, or C, or matlab?"
  A: "I created a data entry form using visual basic for my research project that helped improve the data entry error rate"
-
  Q_teacher: "how well did your interventions work and match your model predictions"
  A: "I didn't get a chance to see as I left the country after the research project (by Warren Buffet foundation) concluded"
-
  Q_teacher: "Shifting gears to python, what are the various data types in python and which one would you use for a table?"
  A: "int, float, list, tuple, dictionary. I'd use a dictionary for a table of numbers."
-
  Q_teacher: "So if the keys were the column names in your dictionary for a table of numbers, what would the values be?"
  A: "A float? (no, it would probably need to be a list or numpy array of floats since), lists, arrays, tuples and dictionaries are called collections because they can contain multiple objects."
-
  Q_student: I will not have too much trouble with the technical aspects. I hope you can help me with motivation and focus and understanding of the US culture since I am new to the country.
  A: "The Springboard counselor and I will certainly keep you on track and help you make continuous progress. Also, I will help you chose among the many technical approaches by recommending the \"best practice\" for a professional data scientist in the US."
-
  Q: "I broke up my number of weeks to complete the project into buckets 0-10, 11-20, ... etc. I created a categorical variable for this with an integer 0-9. How can I use that in my model as a feature?"
  A: "That's not a categorical variable, it's ordinal because it represents a quantity where the order of the categories have meaning, so you can pass it into your model just like any other."
-
  Q: "What other models can I try with this data that I'm using to predict completion of our course?"
  A: "Random Forest may do a little better than logistic regression, but you may be running up against the limit of predictability of human behavior. A better approach would be to try to incorporate additional demographic data about students to create categorical variables that can be used as 1-hot encoded features to input into your model."
-
  Q_teacher: "You're in an interview at Siemens, and they ask you to whiteboard a problem. 'I have the crime dataset for Chicago and I'd like to use it to decide where to put bike racks for our rental U-bike product.'"
  A_student: "You should estimate the crime rate at various locations throughout Chicago and put bike racks in low crime areas."
-
  Q_teacher: "No. Don't ever reccommend a solution. Ask clarifying questions about the data and the problem before recommending a solution."
  A_student: "OK, So what is in the crime dataset? Is it geocoded with latitude and longitude information? Does it have crime type categories? Does it have a category for bike theft? How many crime reports are there? Are there natural language descriptions? Over what time period do you have the reports? Is it live data that is generated every day or static?"
-
  Q_teacher: "What questions can you ask about the target variable and objective of the project?"
  A_student: "What are you trying to optimize? Sales? Theft? Minimize cost? Predict bike thefts?"
-
  Q: "Will discretizing my continuous variable to create an ordinal feature help reduce the noise and error in the output target variable predictions?"
  A: "No, it will not reduce the noise in this feature. Discretizing a continuous variable always reduces information. You should use the raw number in your model whenever possible. The best noise filter is the model itself. Consider a Random Forest or Decision Tree model, they will both discretize all your data with optimized thresholds betwee ordinal classes. And a linear or logistic regression, will average this noise over all the examples in your training set. So discretizing a feature is never a good feature engineering transformation."
-
  Q: "How can I extract the last layer of an InceptionNet v3 for an video action classification model?"
  A: "The terms you want to DuckDuckGo is 'transfer learning keras feature extraction'. That should bring up excellent blog posts like this. The idea is to set the `include_top=False` when you load or instantiate the Keras model. That then allows you to use the `.predict()` method to output a tensor with all the features that were used by the network to predict the class variable it was trying to predict, like classifying images or doing object detection. You need to reshape that tensor into a vector for you new classifier and then train your model using that as your input feature vector instead of the array of frames (images) from the labeled videos in your training set. OpenCV may also have the ability to load Keras and TF models and extract any layer you like during activation. https://www.learnopencv.com/keras-tutorial-transfer-learning-using-pre-trained-models/"
-
  Q: "Should I take the natural language processing course or the deep learning course?"
  A: "In NLP you will learn deep learning concepts like CNNs and LSTMs and you will also learn the specific and complicated tasks of tokening strings and creating word embeddings. These more specialized skills will not likely be covered in the deep learning course. But the deep learning course will have more of the specialized skills for manipulating and processing images (computer vision). Much of your NLP skills will apply to deep learning computer vision problems, but if you are really interested in computer vision the DL class would be a better course."
-
  Q: "Should I transform large dynamic range values, like word frequencies, with the log function so that they display more clearly in a plot?"
  A: "No, you should instead scale the axis with `plot(xlog=True)` or `ylog=True` so that the axis labels reflect the true counts in the data."
-
  Q: "What is wrong with using net revenue as my target variable for predicting movie success."
  A: "The relationship between budget and revenue is usually a nonlinear multiplicative rather than additive. So it would be more informative to train a model to predict the **ratio** of revenue to budget rather than the revenue minus the budget."
-
  Q: "What is a good order to learn NLP concepts."
  A: "The order of the chapters in my book, Natural Language Processing in Action, progresses in a logical incremental way building up from tokenization, to TFIDF."
-
  Q: "What's a good way to learn about the theory behind neural nets?"
  A: "Find a textbook from a university course."
-
  Q: "How many samples do you need to get a **good** estimate of mean or standard deviation."
  A1: "What do you think? What do you think the error in your mean and standard deviation is proportional to?"
  A2: "The error in your standard deviation and mean is proportional to `1/sqrt(N)`. MOst academics will accept a sample size of 30 for calculating a p-value and standard deviation and other critical statistics. But a sample size of 10 is usually sufficient for most practical real world problems. That will usually give you less than 10% error on your statistics."
-
  Q: "I have a model that predicts the adjusted finish time of grayhounds, horses, or people. How can I incoporate information about how long ago it was that that runner raced, because I'm using their previous 5 adjusted finish times as features in my model."
  A: "There's the easy way and the hard way and the super complicated way. The simple way is to compute the average adjusted finish time for each racer over the past N **days** rather than the previous N **races**. The hard way is to add a sample_weights array during training of your model that accounts for how long ago the last race was for each runner. And the most complicated way is to build a separate model that predicts adjusted finish time using **only** the adjusted finish times from the previous 5 races, with sample_weights proprotional to the time since those 5 races. Then you can use that prediction as a new feature to replace the previous 5 adjusted finish times as features in your model."
-
  Q: This dataset of rental bike inspection reports for New York city looks interesting buy I'm not sure how to use it as a time series or geo spatial data.
  A: "First, examine each column of data as if you don't know what the column heading means. How can you turn it into a number? How many numbers/features can you extract from each column? What kind of variable are those numerical features: categorical, ordinal, or continuous. What is the dimensionality of the features generated by each raw column of data? What feature engineering can you do on them to reduce dimensions or cluster them to make them more useful? And finally which column has a target variable (like the inspection report) that has information in it you would like to now before it is available in the dataset? What is the timeline for recording the columns in the dataset. Which values are known before the target variable? Would predicting the target ahead of time have business or scientific value?"
-
  Q: "Should I try to translate a Caffe model to Keras like [this 2-stream model](github.com/wushidonguc/two-stream) to see how much optical flow helps with action prediction."
  A: "That's a great research project or 'future work' idea, but for this project all you need is a hyper parameter table listing all the parameters you tried and the performance as columns. The parameter columns can be things like image_shape/size, seq_length, batch_size, num_neurons layer 1, num_neurons layer 2, etc. The performance columns can be accuracy on your target for the training set and the test set or validation set."
-
  Q: "What plots should I include in my final report."
  A: "The money plot is few scatter plots (pair plots) of the model performance (accuracy) vs the most interesting hyperparameters. So the horizontal axis should be a parameter like batch size or num total neurons or num trainable parameters. The vertical axis should be test set and/or training set accuracy."
-
  Q: "How do I do inferential statistics for Spam classification problem (dataset of spam and ham emails or messages)."
  A: "How did you do it for the discriminatory job interview callback statistics? How did you do it for the human body temperature dataset? You selecte a column of numerical values (categorical, ordinal or continuous feature values) and propose the hypothesis that there is a statistically significant positive (or negative) affect on the target variable. The target can be categorical (like job interview callback or not) or continuous (like body temperature). How could you do that for a dataset where your features are the frequencies (counts) of individual words or tokens and the target is the categorical (class) variable \"is_spam\"? You just pick a word likely to be predictive of spam and calculate the mean in the two classes and run a T-test or P-value test to see if you can discard the null hypothesis. Alternatively you can propose the hypothesis on all the words in your vocabulary in order and then sort them by their p-values to find the most significant predictors of spam. Or you can do dimension reduction (PCA or LSA) or clustering (KNN or LDiA) to generate aggregate classes for multiple words or topics and do your inferential statistics on those words."
-
  Q: "Do you know of any employers in the bay area? I'm interested in the less well-known ones."
  A: "Subscribe to the Owler, SiliconValley.com, silicontap.com, and siliconflorist.com and other newsletters/blogs that regularly report on startups receiving funding. In portland there are [slack](pdxstartups.com), Reddit, and IRC forums where startup news is shared. Meetups and \"Million Cups\" pitches are another great way to get plugged into your startup scene. Whenever you hear of an interesting startup receiving funding or hiring a CTO, visit their website to see if they have job openings in your area."
-
  Q_student: "What is LSA?"
  Q_teacher: "What does the LSA acronym stand for? What do each of those words mean to you? What do you think LSA means when you combine those words?"
  A_student: "Latent Semantic Analysis. Latent means hidden? I'm not sure what semantic means. Analysis means breaking something down into parts or computing numbers that describe the parts of something."
  A_teacher: "You're right. The word 'semantic' means 'meaning'. So semantic analysis is breaking down a natural language document into its pieces of meaning. These are sometimes called 'topics' or 'subjects' when analyzing the meaning of natural language documents. So LSA is an unsupervised machine learning algorithm that creates a vector of topics or meaning for each of a large collection of natural language documents. The 'latent' word is just an expression of researchers facination with the algorithms seeming ability to discover meaning in documents that was hidden from both humans and machines until LSA was invented."
-
  Q_student: "What is PCA and how is it related to LSA?"
  Q_teacher: "What does the PCA acronym stand for? What do each of those words mean to you? What do you think PCA means when you combine those words?"
  A_student: "Principal Component Analysis. \"Principal\" means \"most important\". \"Component\" means \"piece or part of something, often a dimension in a vector\". \"Analysis\" means \"breaking down something into its parts or pieces, usually expressed as numbers for each of the pieces within a vector\"."
  A_teacher: "You're right, and PCA is an algorithm that creates low dimensional dense vectors that express the topics or meaning within high dimensional vectors or tensors like images (light intensity arrays or tensors for each pixel), time series (prices or volume of transactions in vectors), word frequency vectors (bag-of-word vectors), DNA gene frequency vectors, any discrete fixed-length sequences or vectors, song audio sequences (audio amplitude vectors), etc.  When PCA is applied to natural language bag of words vectors (which have thousands of dimensions) it is called LSA (Latent Semantic Analysis) because the vectors PCA produces are semantic vectors that represent the meaning of each document in a lower-dimensional vector."
-
  Q: "Do you know of any data scientist employers in LA? "
  A: "For finance (quant) jobs check out all the banks in Santa Monica. To find lesser well-known companies, subscribe to the Owler and other newsletters/blogs that regularly report on startups receiving funding. In portland there are [slack](pdxstartups.com), Reddit, and IRC forums where startup news is shared. Meetups and \"Million Cups\" pitches are another great way to get plugged into the startup scene in any new area. Whenever you hear of an interesting startup receiving funding or hiring a CTO, visit their website to see if they have job openings in your area."
-
  Q: "Isn't machine learning for optical flow a trivial problem."
  A: "No, it's an active area of research. And if you can come up with some preprocessing, like FFT or optical processing, like microlens arrays, that reduce the latency and computation required it would be extremely valuable in self-driving car and robotics applications."
-
  Q: "The fringe lines produced by a weighted aveage of 2D spatial FFTs should be perpendicular to the optical flow. And L shaped features, like the windows in this self-driving car video should produce X's in the averaged 2D spatial FFT image. So shouldn't I figure out how to combine the two sides of the X to compute the optical flow for that subwindow?"
  A: "The averaged 2D spatial FFT fringe lines look like they are parallel to optical flow to me because they point out radially from the center of the FOV for a camera facing forward on a self-driving car application."
-
  Q: "I have data for water consumption over 15 minute intervals. I also have flags like, 'is_sports_event', 'is_holiday', 'is_weekend', 'is_rush_hour', 'is_business_hours' for each 15 minute interval. And there's a previous 3 hours of precipitation value as well. Some values are nan, particularly in the precipitation and water consumption columns. How can I build a time series model to predict the next day's water consumption?"
  A: "You need to use `DataFrame.ffill()` to fill `NaN` values rather than filling with the mean. Any data preprocessing that brings in future data to the past will help your model cheat. Next, you need to aggregate all the columns for each day using the DataFrame.resample(). Then you can time shift the precipitation water consumption data like this. Here's some fake data with column 'c' as your consumption data. You have to be careful to avoid Pandas' indexing magic which will try to prevent you from shifting your data:
  ```python
  >>> pd.np.random.seed(hash('Peter Watts') % 2**32)
  >>> N = 7
  >>> columns = list('abc')
  >>> df = pd.DataFrame(
          pd.np.random.rand(N, len(columns)),
          columns=columns,
          index=pd.date_range('2020-01-01', periods=N, freq='1d'))
  >>> df = (df * 8).astype(int)
  >>> df
  a  b  c
  2020-01-01  4  4  5
  2020-01-02  4  5  5
  2020-01-03  6  0  3
  2020-01-04  5  3  6
  2020-01-05  6  7  1
  2020-01-06  0  7  5
  2020-01-07  3  6  2
  >>> df['c_tomorrow'] = pd.Series(df['c'].iloc[1:].values, index=df.index[:-1])
  >>> df
              a  b  c   c_tomorrow
  2020-01-01  4  4  5          5.0
  2020-01-02  4  5  5          3.0
  2020-01-03  6  0  3          6.0
  2020-01-04  5  3  6          1.0
  2020-01-05  6  7  1          5.0
  2020-01-06  0  7  5          2.0
  2020-01-07  3  6  2          NaN
  Now you can perform any old machine learning model `.fit()` on the 'abc' columns as your indicator features and 'c_tomorrow' as your target output variable.
  ```
  "
-
  Q: "What is the computational complexity on an unsupervised algorithm to recommend commercial building retrofits to minimize energy consumption? Your goal is to find similar buildings and put them in clusters."
  A: "O(N^2)"
-
  Q: "What is PCA when you apply it to bag-of-word vectors?"
  A: "LSA, Latent Semantic Analysis. PCA.params_ will contain a matrix of word to topic weights that can be multiplied by any high dimensional bag-of-words vector (with that same vocabulary) to produce a low-dimensional topic vector."
-
  Q: "What goes in the presentation slides for the capstone?"
  A: "A summary of all the interesting things you discovered in your report. Visualizations and statistics that you think would be interesting to your audience."
-
  Q: "Because I have mostly categorical variables my correlation matrix with my target variable does not have many continuous correlation values."
  A: "You need to use pd.get_dummies to create one-hot encodings of your categorical variables and you need to encode your ordinal variables and consecutive integers so that `df.corr()` will display all the correlations that you are interested in."
-
  Q_teacher: "An employer asks you to create a recommendation engine for a recruiting firm. They have a database of resumes and job descriptions with some structured metadata for each (job title, location, name, email). They have information about past candidates and the jobs they applied for, interviewed for, and were hired for. What approach would you recommend?"
  A_student: "There are two general approaches for recommendation engines, collaborative filtering and content-based filtering. A content-based approach would use NLP to compare all pairings of job descriptions and resumes to look for good matches. A collaborative filtering approach would cluster candidates and job descriptions and recommend jobs to candidates that were a good match for others in their cluster."
-
  Q_student: "Can I do machine learning on a graph?"
  A_teacher: "Graphs are just data. You can do machine learning on anything. First, think about what a graph is and what kind of data is in a graph database. Then think about machine learning is. What exactly is machine learning?"
  A_student: "Machine learning is using a data predict a new value."
  A_teacher: "Yes but the prediction isn't always in the future in time. Machine learning is using a dataset of example inputs (features) and outputs (targets) to train a machine to write a function that can product similar outputs for similar inputs. Often it's used to predict something new."
-
  Q: "Should I learn `statsmodels.ols` Ordinary Least Squares (OLS) linear regression or is it good enough to use `sklearn.LinearRegression`?"
  A: "The linear regression in `sklearn` is fine for all my work in the real world, but in the medical field or the academic world you might like having all the additional statistics that the `statsmodels` OLS model computes for you automatically."
-
  Q: How do I do a `train_test_split` on time series data?
  A: Your test set needs to be only data that occurs after your training set in time. So your dataset needs to be sorted into time order and you should reserve the last week, month, or year as your test set. The remainder, the first part of your test set is your training set. Do not ever use random sampling with time series data.
-
  Q: "I used get_dummies on my library visitation dataset for my class features, but now I have a huge number of correlations many of which are high. How do I find the correlations that are most useful."
  A: "You are only interested in correlations of your dummy variables with the target variable. So select the column of the `.corr()` matrix corresponding to your target variable and `sort_values()` on it to find the most highly positive and negative correlations."
-
  Q: "Why does the Boston home price linear regression mini project ask me to plot histograms of # of rooms, distance to city center, and student teacher ratio of nearest school to find correlations between them and the home prices?"
  A: "That's not a very useful plot. You would be better off creating a pair plot (scatter plot) of each of your continously valued features relative to the target. If you have any class features you can plot histograms in different colors for each of the class values with home price as the horizontal axis to see how good those classes are at separating/discriminating home prices."
-
  Q: "I have a single fully-connected hidden layer neural network with an FFT preprocessing on a pair of images, with one shifted up to 25% relative to the other. The FFT preprocessor creates a flat 700D vector representation of the pair of images. My batch size 50, by validation size is 400 and my training size is 1600. My regression target variable is the number of pixels of shift in X and Y. After a few epochs my validation loss stops declining at about 0.20 but my training accuracy keeps falling to around 0.05 after 100+ epochs. Is this 'regularization' or 'overfitting' and what can I do about it?"
  A: "Your model is overfitting to the dataset you have created. There are several regularization approaches that can help. What do each of these do and which do you think is most likely to help reduce your validation accuracy? 1. Adding a Random Dropout layer. 2. Reducing the batch size. 3. Adding regularization (L2 or L1 norm of the weights) loss to you cost function. 4. Adding additional data, more random X,Y translations of +/-25%. 5. Doing additional data augmentation (noise, brightness, distortion, rotation) preprocessing. 6. Adding more new source images to your dataset before the cropping and translation and data augmentation."
-
  Q: "I reduced my batch size to 1. Why did this reduce my validation loss?"
  A: "The smaller your batch size the more stochastic your gradient descent becomes. With a batch size of one your optimizer is calculating the gradient for that one sample and estimating the best weights to fit to that image. That will make it adjust the weights in a very random direction relative to the global and local minimum for the optimal weights for all the samples in your dataset. This will help your model find weights that produce a smaller and more general minimum loss for your test set, by randomly exploring your high dimensional space of possible weights. The number of saddle points in your loss function grows exponentially with the number of dimensions. More stochasticity helps your optimizer hop over ridges and saddles."
-
  Q: "How do I do cross-validation and testing of time-series models."
  A: "You need to sort your time-series in date-time order and your validation and/or test sets should be extracted with python slices of data that are in the future relative to all of your training samples. Instead of random sampling for K-folds cross-validation you have to use an expanding window or rolling window cross validation approach so that your final model used for the customer is trained on your entire dataset. Jason Brownlee's [blog post](https://machinelearningmastery.com/backtest-machine-learning-models-time-series-forecasting/) explains it better."
  tags: time series, cross validation, medical, prediction, predictive analytics, digital health, springboard
-
  Q: "Why do I get a `KeyError` when I run the python3 expression `f'{first_name} {last_name}'`. I copied it directly from the _Data Science from Scratch_ crash course in python?"
  A: "Are you using python 3.6+? Have you defined the variables `first_name` and `last_name` prior to instantiating that f-string? The 'f' immediately before the opening quotes defines an f-string or format-string. An f-string expects all template variables within the string (the names between curly braces like `{first_name}`) to be defined in the local namespace prior to instantiating the string so the hidden `str.format()` method can 'interpolate' the string with the values in those variables. This is like the 'r' at the beginning of a string literal which defines a raw string litteral without any string interpolation or escaping of special characters."
  tags: [basic, python, python3, f-string]
-
  Q: "What are the advantages and disadvantages of a Decision Tree relative to a logistic regression or SVM."
  A: "Decision trees are nonlinear and can fit to nonlinear relationships so they can be more accurate for some applications. However, they are less explainable than SVMs and logistic regressors."
-
  Q: "How many interviews should I do before accepting a position? How many offers?"
  A: "It's up to you. Is your first offer a great match for your skills? Do you think you'd be good at it and be passionate about the work? If so, go for it. If you are unsure, then hold out for additional offers. You can delay a decision by asking for more money or benefits or remote work or flexible hours or compute resources (a souped up Ubuntu Data Science laptop). It will typically take a week or more for a company to make a counter offer if your request is out of the ordinary. Human Resources managers and recruiters have significant leeway in what they can offer you without going up the chain of command."
  tags: [career, job search, job market]
-
  Q: "How do I build a time series model that incorporates multiple features other than time?"
  A: "The marine difference in time series modeling and normal models is that you have to respect the order of the dates/times in the dataset when doing a train/test split. Research 'backtesting' or 'time series expanding window cross validation'. You also need to pay attention to time order when you shift your target variable in time to match your prediction horizon. So if you want to predict something at least one day in advance you can shift your target variable by one day so that the target variable for tomorrow is on the same row in your DataFrame as the feature values for today. Features that you know about tomorrow can also be shifted by one day and added as additional features. This works well for features you have an accurate forecast today for a value, like rainfall or max temperature, for tomorrow. Can you think of other features that you know about tomorrow before tomorrow arrives? "
  A_student: "The day of the week, whether it's a weekday, the phase of the moon, the tides, whether it's a holiday, whether it's summer, the number of days since some other event, like the beginning of the dataset or the beginning of some significant event in the news. I need to work on an example problem before I understand it."
  A_teacher: "Stock market data makes good multi variate time series data to model. Predicting tomorrow's stock price or it's price change relative to today. You can even make it a classification model if you create price or percent change ratio ranges of interest, like >1.005 or < .995 or between .995 and 1.005. Also it will give you practice incorporating multiple time series features like daily volume, the high prices, the low price, average news sentiment for the day, or weather. You can create independent ARMA or facebook prophet models for each such time series and use their predicted values as input into your model, if the predictions are accurate. If the predictions are not accurate you can simply use the current day's values for those time series."
  tags: [time series, expanding window, back testing]
-
  Q: "What do I need for a good data science project report."
  A: "The most import thing to have is a table listing all the models and model parameters you tried and how they performed on your training, validation, and/or test sets."
  tags: [hyperparameter tuning, hyperparameter tuning table]
-
  Q: Should I do tokenization before or after TF-IDF calculation?
  A: "Before. Even though the `TFIDFVectorizer` in `sklearn` requires raw text because it uses a tokenizer, it's possible to trick it into using your tokenized text by using `'\t'.join(tokens)` on the input and `tokenizer=lambda: s: str.split(s, '\t')`"
  tags: nlp, tf-idf, TFIDFVectorizer, tokenization
-
  Q: "Should I do TF-IDF calculation before or after `train_test_split()`?"
  A: "After. You need to build your vocabulary for the TF-IDF calculation using only the training set, your test set (and validation set) should use the training set vocabulary and statistics to compute the TF-IDF vectors for the test set: `tfidf = tfidf.fit(trainset); tfidf_vectors_test = tfidf.transform(testset);`"
  tags: [nlp, tf-idf, train_test_split, test set, validation set, vocabulary]
-
  Q: What are all the steps of a typical NLP pipeline for something like spam filtering or sentiment analysis.
  A: "
  1. load the data or instantiate a data generator
  2. clean the data (or instantiate a new generator on the cleaned data)
  3. extract features from your numerical feature (indicator) variables
  4. tokenize the text features in your datasets
  5. extract numerical features from your text features: `len(doc)`, `len([c for c in doc if c in punct])`, `len(tokens)`, etc.
  6. train_test_split to reserve a small holdout test set (10%)
  7. train_test_split to reserve a small validation set (~20%)
  8. BOW vectors for all your text features in all 3 datasets
  9. tfidf vectors for all your text features in all 3 datasets (fit on the trainset so the test and validation sets use the training set vocabulary)
  10. PCA on the tfidf vectors (fit on training set, transform on validation and test set)
  11. token/word embeddings like word-to-vec or glove
  12. document embeddings like doc2vec or Universal Sentence Encoder
  13. model fit (logistic regression, linear regression, NaiveBayes, SVM, CNN, LSTM)
  14. model evaluation: `model.predict` and `.score` (accuracy) on the training validation and test sets
  15. add a row for each model and hyperparameter combination you tried to your hyperparameter tuning table
  "
-
  Q: "I have two datasets from UCI. One with ~120 examples of breast cancer survival with 10 numerical features (blood test results, continuous variables). One with ~156 examples of HCC (liver cancer) survival with ~67 text features, some of which are categorical or ordinal. Which would make the best data science project."
  A: "The HCC dataset would be best for learning datascience because you would have a variety of data cleaning and feature extraction tasks to practice on. But the breast cancer dataset would be much easier to work with because you could immediately fit a simple model on the raw numerical data with only a few lines of code."
-
  Q: "Why does my NLP CNN model for spam prediction get nearly the best training and validation accuracy (93%), but gets 0 false negatives and 0 false positives?"
  A: "Do any of the other models get perfect classification accuracy on the test set?"
  A_student: "No, all my other CNN models and LSTM models get reasonable FP and FN scores."
  A_teacher: "Then you likely have a bug in your code. Did you manually run each model or did you use a GridSearch object or for loop to train and test your models?"
  A_student2: "I manually loaded the saved models for the best ones and copied and pasted the code to compute F1 score."
  A_teacher2: "The best way to avoid bugs and errors is to DRY (do not repeat yourself) your code using a for loop to perform repetitive operations."
-
  Q_teacher: What is overfitting and how do you detect it?
  A_student: Overfitting is when your model doesn't generalize well, it doesn't get good accuracy on new data. You can detect it by calculating the prediction accuracy (like RMSE or F1 score) on the test set and comparing it to the prediction accuracy on the training set.
-
  Q: Based on our retention prediction model, we're planning to do 2 different interventions and compare them to a control group to try to increase retention. Is this a good approach?
  A: It's better to chose your least risky intervention, the one least likely to do harm. Unless you have time pressure, you will get more reliable results if you only do a single small intervention and measure it's effect using A/B testing (comparing to a control group). Then you can spend time examining the results to build on the approach and decide the next best intervention to try. Human behavior is complicated, so you need to minimize the complexity of your causal test protocol.
-
  Q: "How can I use python filter out outliers in a table of home prices."
  A: "Like any programming challenge the first step is to imagine how you would do it manually in your own mind. How would you select the outliers from a table of data if you had to do it by hand using your own brain."
  A_student: "I would plot a scatterplot and look for any values that looked funky and delete them."
  Q_teacher: "Yes but you have to do it mathematically. You can't program feelings. But it turns out there is a score for 'funkiness' of data points of values. It's a very common statistic that can be calulated on any array of values or any distribution of values. Can you guess what the score is? Hint: it's a '____-score'."
  A_student1: "Is it 'z-score'?"
  Q_teacher1: "Yes! Now how can you use a z-score to filter out outliers."
  A_student2: "You could just remove all the point with a z-score greater than 3 (3 sigma)."
  Q_teacher2: "Yes! How do you do that efficiently in python, assuming your data is in a Pandas DataFrame?"
  A_student2: "Using a mask (array of Trues and Falses) to select only those rows within the desired z-score range. The mask goes inside square brackets to select only those rows where there is a True value for the mask, which was created with a conditional expresion like `mask = (df.zscore < 3.0) & (df.zscore > -3.0)`"
-
  Q: "When should I use F-beta score instead of F-1 score?"
  A: "When you know the \"beta\" that is appropriate for your problem. How do you compute the beta for a particular problem? How do you decide what beta to use?"
  A_student: "Beta is the weight or importance of false positives vs. false negatives. So it is determined by how important it is for your model to minimize false positives vs. false negatives."
  A_teacher: "Yes, exactly. For example in a spam filtering problem you want to minimize false positives. If false positives are twice as important in your application than false negatives, then what should you use for your value of beta?"
  A_student2: "Your beta should be 2."
  A_teacher2: "Yes. Now, for a medical diagnostic test for a deadly disease like a blood test for cancer, you would probably like to minimize false negative test results and they might be 10 times as important as false positives. So in that case what should your beta value be?"
  A_student3: "You might use a beta value of 0.1."
-
  Q: "What is a leverage plot?"
  A: "A leverage plot shows the importance or usefulness of each sample in predicting the target variable. It's essentially the magnitude of the z-score for each **sample** relative to the residual. So eliminating outliers is equivalent to eliminating low leverage data points. You can think about it as as the inverse of dimesion reduction. It helps you decide which **rows** in a dataframe should be deleted rather than individual **columns** that should be deleted: http://analyticspro.org/2016/03/07/r-tutorial-how-to-use-diagnostic-plots-for-regression-models/"
-
  Q: "Is the Quora duplicate question detection problem a supervised or unservised problem."
  A: "Supervised because the target variable, a binary label of duplicate or not duplicate is provided."
-
  Q: "What is the dimensionality of the features for the Quora NLP duplicate question detection problem?"
  A: "Very high dimensional. It's the size of your vocabulary if you use a BOW or TF-IDF vectors. So your feature dimensionality could be anywhere from 10s of thousands of words to millions of tokens and n-grams, depending on how you do tokenization and stemming and filtering of frequent or infrequent tokens, such as proper nouns."
-
  Q: "In keras ImageGenerator flow_from_dataframe() takes an x_col argument. Is that the filename?"
  A: "Yes. If that image file does not exist in the directory indicated by the `directory` arg then Keras will throw an exception."
-
  Q: "How is the target variable supplied for `flow_from_directory()`?"
  A: "The target class or value must be encoded in the filename."
-
  Q: In keras ImageGenerator flow_from_dataframe() the y_col argument contains the column name for the target variable. How do I specify a 2-D vector of continuous values for a regressor?
  A: "`y_col` needs to be set to a list string labels for the columns with the 2 values you want to fit to. And class mode needs to be 'other'. For example `y_col=['dx', 'dy'], class_mode='other'`. see https://medium.com/@vijayabhaskar96/tutorial-on-keras-imagedatagenerator-with-flow-from-dataframe-8bd5776e45c1."
-
  A: "The line between supervised and unsupervised learning is often blurry. There's no one right approach. For example, to evaluate an unsupervised learning algorithm you often create a labeled dataset of a few examples in order to evaluate the accuracy of the model."
  A1: "Another example is unsupervised clustering problems. K-Means, for example, has to be told how many classes it should expect, and what number or string to label each cluster with. So you need at least one labeled example for each class, even when you use a classic unsupervised clustering algorithm like K-Means. And you use this to 'learn' the hyperparameters of your unsupervised clustering algorithm."
  A2: "For both classification and regression, PCA is often used as an unsupervised learning algorithm within a supervised machine learning pipeline."
  A3: "On thing to think about is that your target variable for a supervised learning problem can be almost any value associated with the input feature vector or object. For learning word or sentence embeddings your target vector is a portion of the input sequence of one-hot word vectors extracted from the input document."
  A4: "Another thing to think about is that your target variable can be approximate or inaccurate. For example, search engines such as DuckDuckGo rank search results based on the supervised learning target variable of whether user clicks on a particular link or not. For example, if they click on the 9th link on a page, that creates a binary label like 'is_good_search_result'. When that target variable it is used to train a supervised machine learning algorithm that returns search results, it will tend to push links to the top that are relevant to a particular search. This is called 'Learning to Rank' in academic papers."
  A5: "Even though full text search and web search is thought of as an unsupervised problem, you can used supervise learning approaches to solve it. As a Data Scientist building search engines, you will develop a deeper understanding of what's going on will be able to explain it to others. "
-
  Q: "My dataset of clinical medical data contains 150 patient records with 50 features or columns. Many of the parameters contain a large number of NaN or missing values (labeled with '?'). Is this a suitable dataset for doing data science and machine learning?"
  A alt1: "Yes. Most clinical medical datasets have many missing values. Patients are rarely all subjected to the exact same treatment and diagnostic tests, except in medical trials. One approach is to sort your column labels according to the count of NaN values in each column. That way you can start with the columns that contain no NaN values and use those to predict the other columns one at a time, you can use your predictions to 'interpolate the missing values.'"
  A alt2: "Yes. Most clinical medical datasets have many missing values. Patients are rarely all subjected to the exact same treatment and diagnostic tests, except in medical trials. One approach is to directly predict the target variable using only those columns that have very few NaN values. But you will have to drop all the rows that contain any NaN values for your model to be able to work.
  tags: medical, clinical, digital health, dh, ucsd"
-
  Q: "I have trouble understanding the software developement and data science terminology in videos that I watch. What can I do to brush up on my English language skills for software development?"
  A alt1: "You can accelerate your computer science and data science learning by watching youtube videos by people using python and Pandas for DataScience. If there are subtitles in your language that can be a good way to check your comprehension. You want to watch a section of the video without the subtitles and whenever you are unsure of your 'translation' into your own language, go back and rewatch it with subtitles on."
  A alt2: "You can accelerate your computer science and data science learning by attending in-person meetups. Ideally it would be good to test your comprehension of new python and data science terminology in English by answering questions or writing code during or immediately after the presentation. I often keep my laptop open and mimic the presenters commands on my one python terminal, as much as I can."
-
  Q: "Both flow_from_dataframe and flow_from_directory raise similar errors about the inability of Tensorflow or Keras to convert my input_dim specification on the first Dense layer into a TensorShape object."
  A: "What happens if you don't specify input_dim at all? Tensorflow and Keras should be able to figure out the input dimensionality from your generated data."
-
  Q: "What is a leverage plot?"
  A: "A leverage plot shows the amount of accuracy increase in your model prediction for each data point if it is temporarily removed from your dataset during training. So you have to train N models for N datapoints. It's an expensive plot, but can help you identify outliers, points that you can remove so that your accuracy actually improves. It can also indicate some sample_weights you can use to boost the accuracy of a model fitted using those weights."
-
  Q: "How do interpret a leverage plot and use it to improve a model?"
  A: "A leverage plot shows your model prediction accuracy increase for each data point if it is temporarily removed from your dataset during training. You must train N models if you have N datapoints, in order to produce the leverage value for each datapoint in a leverage plot. It's a computationally expensive plot, but can help you identify outliers (low leverage), points that you can remove so that your accuracy actually improves. It can also indicate some sample_weights you can use to boost the accuracy of a model fitted using those weights."
-
  Q: "What is _dropout_ and how does it reduce overfitting?"
  A: "_Random dropout_ is a neural network training technique. A `Dropout` layer can be added anywhere within a deep learning neural network. It randomly blocks the activation of a portion of the neurons in the layer where the Dropout layer is installed. This cause the backpropagation (gradient) calculation to adjust different weights for the same feature values for different samples (examples). That prevents the network from 'memorizing' your training set. It produces a different activation and different weight adjustments for the same input vector during training, spreading the learned information around among many weights. After training, for activation or inference on a validation or test set, the dropout layer becomes a passthrough and does not block any activations. This maximizes the model accuracy."
-
  Q: "I have a jupyter notebook with many different model types. How can I automate the creation of my hyperparmeter table?"
  A: "Create a function for each kind of model you want to evaluate. Create a separate function to evaluate the models with the exact same test set and training set. Create a for loop that iterates through those functions and runs them on your data to populate a list of hyperparameters and accuracy scores, one model for each row in your hyperparameter."
-
  Q: "What is cross validation?"
  A: "Cross validation trains multiple models on different train-test splits of your data. This allows you to compute an average accuracy on your training sets and test sets as well as **confidence** intervals around those accuracy numbers."
-
  Q: "What does a cross validation algorithm return?"
  A: "It does not return a model. Cross validation computes multiple training and test set accuracy values which can be combined into a mean accuracy and a confidence interval on those accuracy values. This is the desired output of CV, the expected accuracy on a new dataset and how confident you can be in that accuracy estimate--how likely your predictions in the future are going to be that accurate."
-
  Q: "What is cross validation used for?"
  A: "Cross validation trains multiple models on different train-test splits of your data. This allows you to compute an average accuracy on your training sets and test sets as well as **confidence** intervals around those accuracy numbers. CV is used to estimate how well your model will work on new unseen data. It's especially useful when you do not have a large training set or your test set is not likely to be representative of examples you will see in the future."
-
  Q: "The accuracy of my logistic regression model on my test set is always exactly 95%, no matter what value of C I use for regularizing the model."
  A: "This tells you that your model is already well-regularized without adding an addition loss term to the model for the number of nonzero weights in your model. Your dataset is large enough and your dimensionality is low enough such that a random sample of your dataset for the test set is always represetative of the other records in the training set (has similar statistics, without any missed outliers). So regularization is not necessary and will not improve your test set accuracy. You are not going to ever have an overfitting problem."
-
  Q: "What is Bootstrapping?"
  A: "You have a sample of measurements and you want to estimate the actually distribution (ECDF)."
-
  Q: "How do you tell if a distribution of values is Gaussian?"
  A: "P-value and T-test score above 0.05."
-
  Q: "What are the future trends in DS and statistics?"
  A: "1. Inferring causal models based on machiine learning models. 2. Topology and Topological Data Analysis (`tdm-mapper`, `sklearn-mapper`)"
-
  Q: "What are some other Regression models besides LinearRegression? "
  A: "Check out the `sklearn` documentation on [Choosing the right estimator](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html). The entire upper right quadrant of the flow chart lists regressors like SGD, Lasso, ElasticNet, RidgeRegression, SVR, and EnsembleRegressors (like RandomForestRegressor). There are many more. "
-
  Q: "What is constrained optimization."
  A: "It's finding the minimum of a cost function like the loss or error model. An SGD algorithm does just that to minimize the error on your training set by adjusting the learnable weights as the parameters of your optimization. If you put limits on the range of acceptable values for those weights, that would be the 'constrained' part of constrained optimization."
-
  Q: "How can I detect cheating and in my model."
  A: "Add noise to your training labels and your accuracy should go down on your test set in proportion to the number of training samples you corrupt with noise. If it doesn't then you have leakage (cheating) happening."
-
  Q: "How does high dimensionality cause overfitting."
  A: "Random samples from a high dimensional dataset will almost always lie within a thin hypersphere shell. The RSS (euclidian length) of high dimenional vectors stays fairly uniform across most datasets. And this length of your vectors is their distance from the origin in your vector space. And all those points will be just as far from the origin as they are from each other. So if you try to build optimal thresholds (bounding boxes or hypercubes) that capture subsets of that data you'll be hard pressed to find boxes that capture meaningful subsets. Your boxes will capture all the points or none of the points. So you must use a model that can find curved surfaces (manifolds) that divide up your space into meaningul chunks for a classifier. For regression the problem is similar."
-
  Q: "What should I do if my dataset has many NaN values. Only 5 of my columns (out of 20) have no nan values."
  A: "Your first model can only use those 5 features, that will be the first entry in your hyperparameter table, the training and test set accuracy on a very poor model that only uses a few features. Then you can come up with ideas for gradually interpolating or imputing values for the NaN values. Starting with the columns that have the fewest nan values."
-
  Q: "How is `pd.get_dummies()` different from sklearn.preprocssing.OneHotEncoder?"
  A: "The get_dummies() function returns a new dataframe containing new columns for all the unique values of your categorical features in your DataFrame. OneHotEncoder() operates on a single array of categorical values so it returns only a single table with columns for the unique values in that array."
-
  Q: "How can I use an unsupervised algorithm to classify email as spam or ham."
  A: "You 'train' your unsupervised clustering algorithm to assign an arbitrary binary label to each email. Then you map those arbitrary labels to spam or ham depending on which pairing gives the best accuracy on your training set. Then you use the accuracy on your test set for that mapping and that model as your true measure of accuracy."
-
  Q: "Name 4 general approaches to building a model to recommend \"friends\" on LinkedIn or Facebook to one another?"
  A: "1. supervised content-based recommendation engine. 2. unsupervised content-based recommendation engine. 3. supervised collaborative filtering model. 3. unsupervised collaborative filtering model."
-
  Q: "Of these 4 kinds of recommendation engines, which is the least likely to provide an accurate model? 1. supervised content-based recommendation engine. 2. unsupervised content-based recommendation engine. 3. supervised collaborative filtering model. 3. unsupervised collaborative filtering model."
  A: "3. A supervised collaborative filtering model will reuse the collaborative filtering label (match or not match) and will thus not produce any better results than a hard-coded unsupervised collaborative filtering model built on cosine distance metric or euclidean distance or any other emperically designed distance matrix."
-
  Q: "A coding challenge asked me build a binary classifier but the dataset is biased because it has more positive than negative examples (more ones than zeros). My first attempt was to make the number of 0 labels equal to the number of 1 labels by truncating the dataset. What else can I do with my SVM?"
  A: "First that's called an 'imbalanced' dataset not a 'biased' dataset. DuckDuckGo for 'sklearn imbalanced training set' to get ideas. Your first approach is called 'undersampling the majority class.'  Which class would you oversample in order to maintain a balanced training set? How would you accomplish that oversampling? Are there some other approaches?"
  A_student: "I could oversample the minority class using sample with replacement (pandas.DataFrame.sample()) on a dataset of only positive examples. I'd alternate between tables containing the minority and majority classes with equal probability during that sampling. Another approach might be to use the `class_weights='auto'` argument in many classifiers like SVC."
  A_teacher: "Yes, that would work well. But a better oversampling approach is to shuffle the two tables and build a generator that perpetually iteratest through each class (each table) reshuffling and restarting the iterator whenever it reaches the end. That will ensure all of the samples are reused an equal number of times, maximizing your learning rate. And yes, `class_weights='auto'` is the most efficient way to train a classifier on unbalanced datasets."
-
  Q: "What are word vectors and what are they used for?"
  A: "Word vectors are a vector representation of the semantics (meaning) of words. They can be used in algebraic expressions like `wv['queen'] - wv['woman'] + wv['man']` to perform reasoning and inference. This example expression would produce a word vector that is very close to the word vector for 'king' (`wv['king']`). They can also be used in any pipeline where one-hot encoding of words into vectors are used. They reduce the dimensionality of your vector encodings of words but they make the encodings dense and continuous. This is a transfer learning technique for natural language processing pipelines, typically used for LSTM or RNN models of word sequences."
-
  Q: "What is an efficient way to define a set of hyperparameters to train and test a machine learning pipeline?"
  A: "The pipeline should be parameterized as a function with arguments that control things like the regularization coefficient (`C`) in a logistic regression. Even the model type and name should be parameterized (e.g. `sklearn.LogisticRegression` and `sklearn.NaiveBayes`). Then you can call that function with those parameters defined in a kwargs dictionary, e.g. `kwargs = dict(model_class=sklearn.LogisticRegression, C=0.1)`. You can update a default hyperparameter dictionary with new parameters for each step in a loop that iterates through a list of dictionaries of hyperparameters. These need only contain the changes to the default hyperparameter kwargs dictionary."
-
  Q: "How can I reduce overfitting for an XGBoost model?"
  A: "How do you reduce overfitting for other similar models?"
  A1 (student): "For random forest I increase the number of trees and reduce the depth of the trees. But for XGBoost increasing the number of trees may increase overfitting."
  Q1 (teacher): "You are exactly right, so the best way to learn is to try adjusting each of those hyperparameters on a a toy problem or a real problem you are working on that trains (and overfits) quickly. What are some other overfitting techniques you can use?"
  A2 (student): "Regularization, feature selection, and dimension reduction. But does XGboost or random forest have a regularization parameter in scikit-learn?"
  A2 (teacher): "I don't know. You'll have to read the docs."
-
  Q: "When reading the Enron email dataset I only read the first directory of spam emails and the 200th email caused a Unicode error when reading the file."
  A: "You need to open and read the file in binary mode as a bytes array instead of a UTF-8 string. Then you can decode the bytes array using `.decode('latin')`. Alternatively it may be possible to open the file in text mode using the 'latin' decoder. Don't forget to use `.splitlines()` rather than `.split('\n')` because some e-mail will use the old Windows or Mac line endings to delimit lines. "
-
  Q: "What are some good questions to ask an interviewer in an informational interview?"
  A: "As a data scientist what do you want to know about the company or business or people?"
  A_student: "How easy they are to get along with. How interesting is the project they'd have me working on, etc."
  Q_teacher: "Yes, in addition what does a _data scientist_ focus on in their work? What matters most to their success at work? What is a data scientist most passionate about? What is the key ingredient for a data science predictive model."
  A_student: "A data scientist is mostly intersted in the data and the processes around it. What is the quality and amount of data for the problems the business has? How much is management behind the use of data to make decisions? Do executives pay attention to quantititave metrics when they are making decisions (data driven company) or are they focused mostly on their influence and power to control the organization using their own brilliant intuition? Ego-driven software development decisions or evidence-based decisions? You want to work for a company where data science is at the core of what they do. Otherwise you will spend your day trying to convince others to pay attention to what your models are telling you about their business."
-
  Q: "Why is my model getting 100% accuracy (`.score() == 1`)?"
  A: "You have probably passed in the `y_pred`, the predictions for your model, rather than `y_test` the True values of your target variable. So that score is just confirming that your model is perfectly correlated with the `X_test` -> `y_pred` mapping that you created by running `y_pred = model.fit(X_test)`.  That just means that your model is not stochastic. It produces the same predictions whenever you run it on the same input features. So you should do `model.score(X_test, y_test)` instead of `model.score(X_test, y_pred)` to get a real measure of your model's performance. "
-
  Q: "How does correlated feature affect random forest and feature importance?"
  A: "Design a small experiment to find out?"
-
  Q: "How to overcome overfitting in boosting?"
  A: "What are some techniques for overcoming fitting in general? Will they work with boosting models?"
-
  Q: "How to deal with missing information. For example, let's say Uber has customer information and want to build a model to predict user retention. In the data set, 1/2 of user rating for driver is missing. Intuitively, there are reasons why user choose not to rate a driver. It's more likely that not leaving a rating means they are not satisfied with the trip. So instead of fill in mean/0 in the column, can I create a separate feature to indicate whether a user leaves rating for driver, but I still need some numbers in the user rating column to train my model, should I use mean/0 fill in that column?"
  A: "Yes, you should always use judgement when interpolating or filling data and your goal should be to retain as much information as possible. so the first thing to try for any categorical value is to create a new category for 'missing_value'. The equivalent of that for an ordinal or continuous variable is to assign is to create a new feature for `missing_feature_x` like you suggested. You will also need fill the value with an appropriate continuous value (like the min max or mean or an inferred value based on an ML model like linear regression on your training set where your target becomes one of your features and the missing value feature becomes your target, and you can only train on examples where the value is not missing). Inferring the value from the other features is the most accurate and least destructive. If you have many continuous or ordinal features that have missing values, you need to start this process on the features with the fewest missing value samples."
-
  Q: "Let's say in real world, our data is imbalanced. For example user rating for a resturant have far more 5 stars than 1star, or for example fraud has far less 1 than 0s. When we train our model, we use sampling method to make class balanced. How do we expect its performance in real world? Do we see any problems of deploying a model trained on balanced data and used on real world?"
  A: "Your test set performance should tell you what the real world performance will be, always. so you can test any hypotheses you have by running experiments and seeing how it affects test set performance."
-
  Q: "Where can I find a corporate e-mail corpus?"
  A: "The Enron email dataset has > 100k emails, many of them classified as spam/ham or given other tags and category labels."
-
  Q: "Should I apply get_dummies to my test set as part of my inference (prediction pipeline)?"
  A: "No. You should not use `get_dummies` for a production pipeline that needs to work on a new unseen dataset, such as the testset. You should use `sklearn.feature_extractors.Vectorizer`. New unseen data may not contain all the same category values as your training set, or the categories might be in a different order. That would cause your feature vector become nonsense to your model. It might raise a `ValueError` because your vectors had the wrong number of dimensions. Or, even worse, it might run just fine, but produce horrible predictions, because the columns are in a different order."
-
  Q: "What is a multitask model?"
  A: "A multitask model is one that is trained to solve multiple different tasks. A model that solves a multi-label problem, like an image tagger is one example. A tagger might have tags for indoor/outdoor and also tags/categories for cat/dog/person/tree. That would be a 2-task model. The challenge with any multitask model is to make sure your back propagation (or gradient descent) ignores the gradient for labels that are missing in your training set. So if you have an image that is labeled for indoor/outdoor but not cat/dog/person/tree you should flag the missing labels with -1 in the 4 binary labels positions for the missing labels. Your loss function can then ignore those labels when computing gradients by masking them out from the error calculation."
-
  Q: "How can I build an unsupervised model to detect when people are lying about the school they attended on their resume?"
  A: "A good unsupervised model might use a graph theory algorithm called shortest path search. [Dijkstra's algorithm is one example](https://en.wikipedia.org/wiki/Shortest_path_problem#Algorithms) For each school, you'd have to do N^2 (N=number of people who claimed to have attended that school) shortest path searches to estimate the fakers for that school. You'd then have a distance for each person to each other peer that claims to have attended the same chool. You could aggregate all these minimum path lengths with the min, max, median or mean. Then you could calculate the likely number of modes in the distributions for each school. If it's bimodal, you The portion of the distribution that contains the larger distance to their peers at the school is likely to contain many of the 'fakers'."
-
  Q: "SVM took more than 40 minutes to train on my natural language problem. That was surprising. I had to cancel the training before it finished. What should I do?"
  A: "It's good to develop an intuition about how a particular model scales as you add more data. The best way to do that is to Train on a much smaller training set. You can then gradually increase the size of the training set (doubling it) to see how that affects runtime. From that you can estimate if it's an O(N) or O(N log(N)) or O(N^2) (polynomial) algorithm or something in between. The scikit learn documentation on 'Choosing an Estimator' contains a valuable flowchart that includes selection criteria for the dataset size, based on which models scale well to large datasets. However, you should only look at this chart after you've done your own experiments on your own problem/data. You may be able to develop a more nuanced datascience intuition than what the very general guidelines with broad assumptions about the nature of your data and problem."
-
  Q: "What is causing the unhashable slice object on this line in my code: `subplot_ax.plot(x=X[:,2], y=X[:,3])`?"
  A: "What kind of object is `X`?  It's probably not a numpy array or any other kind of object that returns an array of floats when sliced. It could be a `dict` or `DataFrame` object which both rely on the key between square brackets to be a hashable type like a `str`,`int` or `tuple`."
-
  Q: "How can I reduce the training time on an NLP processing problem? My LogisticRegression on 5,000 TFIDF vectors is taking forever!"
  A: "For any ML problem reducing your the number of features your model has to fit to will have a large reduction in training runtime. For NLP this means reducing your vocabulary. Eliminating stopwords and lematization are often done, but they won't help nearly as much as increasing min_df to above 5 or so. Without 5 or more documents that mention the token or n-gram in your vocabulary, your model will not be able to learn much about it's usage statistics anyway."
-
  Q: "What is a good plot to use for tuning hyperparameters?"
  A: "First you should sort your table of hyperparamers by the test set accuracy. Then you should plot one of the more interesting quantitative (ordinal or continuous) hyperpameters on the horizontal axis with testset accuracy on the vertical axis. Some interesting hyperparameter is the number of trained parameters or the number of features extracted from your data set (vocabulary size for a NLP problem)."
-
  Q: "What is the best plot for finding the optimum hyperpameters?"
  A: "Like any multivariate optimization problem you may have to plot each of your variables (hyperpameters) separately on the horizontal axis and then plot a scatter plot with the value you're trying to optimize (usually model accuracy or loss) on the vertical axis.  If think there might be interactions between you hyperparameters, you can try to plot a 3D contour or surface plot with those two hyperparmeters on the two horizontal axes (X and Y) and the optimization target value (accuracy) on the 3rd axis (Z)."
-
  Q: "Why do I need to keep track of the experiments I run? Can't I just delete the code when I find that it doesn't improve model accuracy?"
  A: "Every experiment is a leaning opportunity. If your hyperparameter table contains all the combinations hyperparameters you've tried along side the accuracy you achieved you'll be able to develop an intuition for what helps improve accuracy for a particular kind of problem. This is called the hyperparameter terrain. Many of the hyperparameters interact with each other in unexpected ways to improve or reduce accuracy. It's not possible for a human mind to accurately recall all these interactions. And people who read your report may ask you if you tried a particular combination. Your future self will appreciate it if you've recorded your results (and even the code) so you can answer questions about what you've done and what you've learned."
-
  Q: "Should I use `pd.DataFrame.get_dummies()` to convert all my `np.object` type columns (strings) into numerical values for machine learning?"
  A: "No. You need to use a scikit learn Vectorizer in order to ensure you have separate `.fit()` and `.transform()` stages within your pipeline. If you do `get_dummies()` on your entire dataset you will create columns based on the testset data, which would be cheating. Some categorical values might not exist in your training set once you do train_test split. And even if you wait until after `train_test_split()` to do your `get_dummies()`, it will be difficult for you to transform your test set into the exact same vectors of binary values for each of your categorical features."
-
  Q: "Should I use `pd.DataFrame.get_dummies()` to convert all my `np.object` type columns (strings) into numerical values for machine learning?"
  A: "No. You may unnecessarily create too many binary features. Your ordinal features may be listed as `object` types in your `DataFrame`. For example, age might exist as a string like '21 yrs' in your database. You should convert it to an ordinal variable by stripping off the years and converting units if necessary (for strings like '18 years and 6 months') so that your model can generalize from this age value and more accurately predict your target variable."
-
  Q: "You created a model based on 3 different approaches for filling missing (NA) values using the `.fillna()` function on your dataframe. What is the minimum number of columns and rows you should have in your hyperparameter table?"
  A: "Your hyperparameter table should have at least 3 rows, one for each of your experiments with `min`, `max` and `mean` for `fillna()`. Your hyperparameter table should have at least 3 columns as well. You need one column for the one hyperparameter that you varied, called 'fillna method', and it should have values of `min`, `max` or `mean`. You should have at least two more colums to show the accuracy or performance of your model, typically 'testset F1 score', and 'training set F1 score'."
-
  Q: "What are the column names in a hyperparameter table?"
  A: "The names of the hyperparmeters you varied from experiment to experiment, as well as the names of the performance variables that you calculated to evaluate model performance, such as 'test and training set F1 score'."
-
  Q: "What are the row names in a hyperparameter table?"
  A: "The names, ID numbers, or file names of the experiments that you performed. Each experiment should change at least one hyperparameter."
-
  Q: "Is the train test split ratio a hyperparameter?"
  A: "Yes, but it's not a very interesting one. You should not vary it between experiments. It should remain fixed throughout all your experiments. If you change it, the performance of your model will not improve or degrade significantly."
-
  Q: "The colors in my legend in my violin plot don't seem to be correct. What should I do?"
  A: "You can just write the text for the legen in your report or notebook as a label or comment on the plot."
-
  Q: "If you trained NLP models on TF-IDF vectors with a `max_df` of  .11, .35, and .55 which one would you expect to overfit the least?"
  A: "A TF-IDF `max_df` parameter of .11 will overfit the least."
-
  Q: "There's a Kaggle competition to predict the 'adoptability' of pets at an animal shelter. Each pet (cats and dogs) are rated on a 0-4 scale representing how the amount of time it took for them to be adopted. It includes pictures of the pet and 15 features like gender, breed, cat or dog, weight, height, any injuries or medical issues. What do I need to find out to decide if this is a well-formed Data Science problem?"
  A: "Whenever you're considering a new data science or machine learning problem, the first question you should ask yourself is how many samples or examples are there (number of rows in the data table). This alone doesn't tell you much about the problem. You need to compare this number to the total dimensionality of the features you are using to predict the target variable. In this case the image data could have more than a million of pixel values (millions of dimensions) for each record, so this will be a difficult problem if you want to use that image data to solve the problem. 1000 pictures or examples would likely be the bare minimum you need to have a chance of using the image data to solve this problem."
-
  Q: "How many records, examples, or samples do I need for a good data science or machine learning dataset?"
  A: "It depends on the number of features and the overall total dimensionality of those features. If your feature dimensionality is larger than the number of examples you have then your problem is likely a deep learning problem. It's unlikely you'll be able to use conventional models like linear regression, logistic regression, or even random forests to solve such a problem. And you'll need many examples (much more than 1000) to solve most problems that require a deep learning network."
-
  Q: "In the expression `from scratch import *` what is `scratch`?"
  A: "It's a package or module name. You can find it on the [Cheese Shop](https://pypi.org/project/scratch) by [`duck`](http://duck.com)ing 'install scratch python package'. The scratch package is for Joel Grus's book _Data Science from Scratch_ which contains a python crash course as well as some fun data science exercises you work without install `scipy`, `scikit-learn` or any of the other big Data Science python frameworks."
-
  Q: "How do you fix this bug: `ModuleNotFoundError: No module named 'scratch'`?"
  A: "What does the error message tell you that `scratch` is?"
  A_student: "It's a module."
  A_teacher: "Yes it's a module once you import it into python, but it's also a package outside of your environment that you can install by using the command shown at the top of the [docs page on pypi](https://pypi.org/project/scratch). If you're in the console or shell: `pip install scratch`. If you're in an ipython or jupyter console: `!pip install scratch`."
-
  Q_student: "I compiled opencv on raspbian stretch on a raspberry pi B 3+ but I can't import opencv."
  A_teacher: "Are you using a virtual environment?"
  Q2 (student): "I didn't think that was important. I skipped those steps."
  A2 (teacher): "Make sure you activate a virtual environment before you pip install opencv. Then you need to make sure you activate that same environment before launching python. You can tell which virtual environment you are in by typing `which python` to show the path to the virtualenv. And you can tell the version of python by typeing `python --version`. Make sure it's python3.5+."
  date: 2019-02-28
-
  Q: "Why can't I connect to my RDS PostGreSQL instance from my AWS EC2 instance. I can connect from other IP addresses and EC2 instances."
  A: "You need to make sure egress on port 5432 is allowed. AWS instances do not allow all outgoing ports by default, you have to explicitly turn them on."
-
  Q: "In the expression `from scratch import *` what is `scratch`?"
  A: "It's a package or module name. You can find it on the [Cheese Shop](https://pypi.org/project/scratch) by [`duck`](http://duck.com)ing 'install scratch python package'. The scratch package is for Joel Grus's book _Data Science from Scratch_ which contains a python crash course as well as some fun data science exercises you work without install `scipy`, `scikit-learn` or any of the other big Data Science python frameworks."
-
  Q: "How do you fix this bug: `ModuleNotFoundError: No module named 'scratch'`?"
  A: "What does the error message tell you that `scratch` is?"
  A_student: "It's a module."
  A_teacher: "Yes it's a module once you import it into python, but it's also a package outside of your environment that you can install by using the command shown at the top of the [docs page on pypi](https://pypi.org/project/scratch). If you're in the console or shell: `pip install scratch`. If you're in an ipython or jupyter console: `!pip install scratch`."
-
  Q: "How can I improve the accuracy of my real estate home price estimator?"
  A1: "You can fill your NaN values in your feature vectors with predictions for each of those values based on all the other features for which you have complete data. This is called interpolation, or \"imputing\" data. You want to start with the feature that has the fewest NaN values, then work your way up to the features that have many NaN values. As usual, each time you impute a new set of values you should retrain your model and see how it affects your test set accuracy. And you should record each of these training runs in a hyper parameter table."
  A2: "You can also try different models like random forest regressor, a polynomial regressor, and an artificial neural network."
  A3: "For each model, or data cleaning approach, or hyperparameter used to configure a model, you should record a new row in a table or database that lists all the pieces of your pipeline and how they were configured differently from the other models in your table. This is called a hyperparameter table. You can use it for inspiration to find additional hyperparameters to adjust, or parts of the hyperparameter space to explore. You should save your code for each pipeline (each row in the hyperparam table) so you can recreate the results if asked."
-
  Q: "What do I do about NaN or missing values that appear to actually mean something, like \"fence type\" for a realestate home price estimator."
  A: "In your example, a `fence_type` of NaN or None or '' probably means the home does not have a fence. In this case you should replace it with an integer like 0 or a string like 'None' to represent this does not have a fence categorical value."
-
  Q: What is Keras? Was it created by Google?
  A: Keras is a high level framework for building deep learning models. It abstracts away the details of several graph computation engines so you don't have to learn their particular syntax. You can use Theano, TensorFlow or even MXNet as your backend engine for Keras. And Keras has both a functional API as well as a Object-Oriented API.
-
  Q: "How can I automate my hyperparameter tuning if all my models are in a single jupyter notebook on Google collaboratory?"
  A: "If you select `download .py` from the File menu in your notebook, you can convert the entire notebook into a single function that returns your accuracy metrics on the training and test set. If your function returns a dictionary you won't have to remember the order of the returned values. You can then upload the edited py file back to collaboratory and add these 2 lines at the top so you can import it from a new jupyter notebook: `from google.colab import drive;` `drive.mount('/content/gdrive')`. Your hyperparameters become the arguments to this function, so you can import it and then iterate through a list of dictionaries of hypterparameters and then put all the results `dict`s into a `DataFrame` for sorting and display in your report."
-
  Q: "What more do I need to add to the conclusion section of my data science report?"
  A: "In addition to summarizing the results shown in your hyperparameter tuning table, ask yourself what insights you can glean from that table? Which model and hyperparameters performed best you want to generalize your learning from the hyperparameter tuning table? Do you think this is generally true for similar problems? Review the table to see if you see any patterns. Did some models overfit more than others? What do you think the best model will be if you found the perfect hyperparameters? Is there some structural difference in the way you did data cleaning or the kind of model that produced better results? Are there some hyperparameters that helped improve training set accuracy or test set accuracy across the board? For example increasing the ngram length from 1 to 2 almost always improves accuracy on the training set and sometimes improves accuracy on the test set, for models that generalize well using regularization."
-
  Q: "When I compose a data science presentation should I include a lot of technical detail for an audience of both data scientists and managers and lay people."
  A: "Your presentation should contain enough statistics and visualizations for both audiences. Presentation slides are meant to be presented by you verbally. You can go into more detail on the plots or statistics that would be of interest to which ever parts of your audience you want to address."
-
  Q: "I know how to create a `for` loop that iterates through different scikit learn models and use different hyperparameters for my models, but the data cleaning and normalization stages do not have hyperparameters. How do I do hyperparameter tuning on the data cleaning and normalization steps. "
  A: "Actually, every argument to every function call in your pipeline, including the default arguments can be thought of as hyperparameters for your model. After all, they will each affect the accuracy of your model. So you just need your `for` loop to iterate through different versions of you entire pipeline, including data cleaning and adjust some of the data cleaning hyper parameters in order to find out which pipelines give you better accuracy. The most reliable and reusable way to parameterize the entire pipeline is with a `sklearn.pipeline.Pipeline` object. However, if you have transformations and data cleaning steps that are custom functions, without `.fit()` and `.predict()` or `.transform()` methods then you will have to create a class definition derived from the `sklearn.BaseEstimator` class and add those methods that call your custom cleaning and transform functions from within those class methods. Hnyk has [a good tutorial](http://danielhnyk.cz/creating-your-own-estimator-scikit-learn/ on how to do this. This requires an understanding of Object-Oriented programming."
-
  Q: "How can I automate my hyperparameter tuning if all my models are in a single jupyter notebook on Google collaboratory?"
  A: "If you select `download .py` from the `file` menu in your notebook, you can convert the entire notebook into a single function that returns your accuracy metrics on the training and test set. If your function returns a dictionary you won't have to remember the order of the returned values. You can then upload the edited py file back to collaboratory and add these 2 lines at the top so you can import it from a new jupyter notebook: `from google.colab import drive;` `drive.mount('/content/gdrive')`. Your hyperparameters become the arguments to this function, so you can import it and then iterate through a list of dictionaries of hypterparameters and then put all the results `dict`s into a `DataFrame` for sorting and display in your report."
-
  Q: "What more do I need to add to the conclusion section of my data science report?"
  A: "In addition to summarizing the results shown in your hyperparameter tuning table, ask yourself what insights you can glean from that table? Which model and hyperparameters performed best you want to generalize your learning from the hyperparameter tuning table? Do you think this is generally true for similar problems? Review the table to see if you see any patterns. Did some models overfit more than others? What do you think the best model will be if you found the perfect hyperparameters? Is there some structural difference in the way you did data cleaning or the kind of model that produced better results? Are there some hyperparameters that helped improve training set accuracy or test set accuracy across the board? For example increasing the ngram length from 1 to 2 almost always improves accuracy on the training set and sometimes improves accuracy on the test set, for models that generalize well using regularization."
-
  Q: "Tab completion and history search with ([ctrl]-r) doesn't work on my windows machine when I run `ipython` or `jupyter console` in the git-bash terminal (MinGW64). What should I do?"
  A: "Add an alias to your .bashrc so that your `ipython` console is wrapped with the `winpty` terminal emualator for windows: `$ echo \"alias ipython='winpty ipython'\" >> ~/.bashrc`."
-
  Q: "My rsync command isn't working. I can ssh into the server and I've set up my `~/.ssh/config` file with a Host entry for the server that uses the correct `Port`, `User`, `IdentityFile`, and `Hostname` settings. I don't have to enter a password when using ssh to that hostname."
  A: "Your rsync command should not include any arguments except the paths to the files you want to sync. The label of the `Host` variable in your config file should be all you need to specify the server within your path. Does your working ssh command include a `-i` (IdentityFile) argument?"
  followup: rsync 2)
-
  context: rsync followup 1
  Q: "No."
  A: "OK. Does your working passwordless ssh command include an @ symbol? Are you specifying a username before the `@` symbol for the host, FQDN (Fully Qualified Domain Name), or IP address? Examples that specify the username might be `username@fully.qualified.domain.name` or username@127.0.0.1."
  followup: rsync followup 2
-
  context: rsync followup 1
  Q: "Yes."
  A: "OK. So delete the `-i` argument from your `ssh` and `rsync` commands and see if they work. They both should work with only the paths (including the `Host` label from your `~/.ssh/config` file) that you want to connect to. Make sure the `IdentityFile` (path to your private RSA key file) is specified in your config. Under the `Host` label you are connecting to you should see a config variable called `IdentityFile` with the same path you used in your ssh command as the `-i` parameter."
-
  context: rsync followup 2
  Q: "No."
  A: "OK. Does your working passwordless ssh command include an @ symbol? Are you specifying a username before the `@` symbol for the host, FQDN (Fully Qualified Domain Name), or IP address? e.g. `username@fully.qualified.domain.name` "
  followup: rsync followup 3
-
  context: rsync followup 2
  Q: "Yes."
  A: "OK. So delete the `username@` in your `ssh` and `rsync` commands and see if they work. They both should work with only the paths (including the `Host` label from your `~/.ssh/config` file) that you want to connect to. Make sure the `User` (linux username on the server) is specified in your `~/.ssh/config`. Under the `Host` label you are connecting to you should see a config variable called `User` with the same username you used in your `ssh` command before the `@` symbol."
-
  context: rsync followup 3
  Q: "No."
  A: "OK. Does your working ssh command include a `-p` (`Port`) argument? For example: `-p 22`."
  followup: rsync followup 4
-
  context: rsync followup 3
  Q: "Yes."
  A: "OK you need to delete the username and the `@` symbol from your `ssh` and `rsync` commands and make sure the username is specified under the appropriate Host section in your config as a `User` variable."
-
  context: rsync followup 4
  Q: "No."
  A: "OK. Are you using the same name for the `Host` from the `~/.ssh/config` file in your ssh command? For example if your host file says `Host servername` and under that it says `Hostname servername.company.com` your ssh command should be `ssh servername` *not* `ssh servername.company.com`."
  followup: rsync followup 5
-
  context: rsync followup 4
  Q: "Yes."
  A: "OK. Then you just need to replace the host label in your ssh command with the `Host` config value and see if that works. If not you need to follow [the procedure](https://blog.tinned-software.net/ssh-passwordless-login-with-ssh-key/) for setting up passwordless login in your `~/.ssh/config` file."
-
  context: rsync followup 5
  Q: null
  A: "If `ssh` and `rsync` still don't work, I'm out of ideas. You may have a permissions problem on your `~/.ssh/` directory on your client computer where you are issuing the `ssh` and `rsync` command. Or problems with `/etc/ssh/sshd_config` on the server (`HostName`) you are trying to connect to."
-
  Q: "In an `sklearn` `KNeighborsClassifier` (k-means) model what is K, and what value should I use for it?"
  A: "K is a hyperparameter. It is the number of neighbors whose class label is averaged or voted together to infer the label for a new sample. The only way to find the optimal value for K is to perform hyperparameter optimization. Hyperparameter optimization is when you record the test set accuracy for many possible values of the hyperparameter (`K` in this case) until you can be reasonably confident that you've found the best value for your problem."
-
  Q: "What additional useful piece of information does K-folds cross-validation give you that test set performance evaluation does not?"
  A: "Test set performance evaluation gives you an estimate of the accuracy of your model on future data. Cross validation gives you a confidence interval on that expected performance."
-
  Q: "How can i do K-folds cross validation on time series data?"
  A: "You must obey the time order of your samples when spliting your data into training and validation sets (folds) during cross validation. The best way to do this is with 'sliding/rolling window' cross validation, or even better ['expanding window' cross validation](https://machinelearningmastery.com/backtest-machine-learning-models-time-series-forecasting/) rather than 'K-folds' cross validation. Otherwise `K-folds` will split your data randomly allowing future data to leak into the past, giving your better performance than you can expect in real life where time moves forward and not backward ;)."
-
  Q: "I have 7 features describing how often students interact with the website and how much of the assignments they've completed in the past month. Which clustering algorithm should I use (KNN, K-Means, DB-Scan, etc) and what can I use the cluster information for?"
  A: "The categorical label found by a clustering algorithm may or may not represent anything meaningful or useful about your problem or the things you want to be able to predict, like completion rate for students. The only way to find out is to add the cluster ID as another feature to a machine learning model, that is attempting to classify or regress to a target label you are interested it. A more common approach is to use clustering on demographic data to identify segments of your customers so that you can target your product features and your marketing campaign for difference segments of your market. The Scikit-Learn documentation has [a nice table](https://scikit-learn.org/stable/modules/clustering.html#overview-of-clustering-methods) showing the advantages and disadvantages of 10 popular clustering algorithms. You should choose one that looks like it would work will with your data and will run in a reasonable amount of time for the dimensionality and size of your dataset."
-
  Q: "How do I create a folder within my github repository?"
  A: "If you learn how to use the command line tool you can push any file or directory (that isn't empty) using the standard `git add`, `git commit`, `git push` workflow. If you'd prefer to use a GUI you can download http://desktop.github.com for your laptop and connect it to your github account so you can clone and manage any repositories there."
-
  Q: "What sort of hyperparameters should I try?"
  A: "You can try different models, and model types, like random forest regressor, linear regression, polynomial regression."
-
  Q: "How do I do polynomial regression?"
  A: "The easiest way is to transform augment your features with additional features that are the polynomial exponents that you want to try, by say squaring all the features so that you have twice as many features as your started with."
-
  Q: "What is the difference between a Senior Business Intelligence Analyst and a Business Analyst?"
  A: "The ranking of data science postions in an organization like Amazon, in order from most experienced to least experienced would be: '''
      1. Senior Data Scientist--capable of applying cutting edge models to new problems (e.g. Kaggle competition top 10-20%
      2. Data Scientist -- able to develop software for best-in-class models for applications that have previously been solved using similar models (e.g. Kaggle competition middle of the pack).
      3. Senior Business Intelligence Analyst -- not a CS major but proficient at a high level language like R or python. Proficient at complex SQL queries like temporary intermediary tables and graph queries. Capable of producing visualizations and statistical measures of the performance of predictive models. Can create plausible hypotheses and formulate strategies for testing them.
      4. Data Analyst or Business Analyst -- not proficient at software development but can manipulate and query databases using basic SQL and VB scripts or formulas in Excel. Can use commercial tools for data visualization and plotting.
    '''"
-
  Q: "A data science bootcamp instructor claimed that some models *require* input features and target variables be normally distributed? Which models are these?"
  A: "This is false. No models *require* normally distributed data. This may come from a misunderstanding of the textbook descriptions of models like Linear Regression. Some models *assume* normally distributed data in order to make it possible to prove, theoretically, that they are optimimum in some way. This usually means that some particular measure of error is minimized or some accuracy metric is maximized. However, this does not in any way mean that the input must be scaled with something like the NormalScaler or that if it is skewed or uniform or in any other way nonnormally distributed that it will not work well for that model. In fact Ridge and Lasso regression define cost functions for which this normal data assumption is not even applicable. And Ridge and Lasso work better on almost any dataset and any problem than linear regression alone works."
  Q: "A data science bootcamp instructor claimed that some models *require* input features and target variables be normally distributed? Which models are these?"
  A: "This is incorrect. No models *require* normally distributed data. You can discover this yourself with any nonnormally distributed dataset you like. You will find that sklearn.LinearRegression and other models that are *designed* for normally distributed data will usually work equally well whether your scale the data to be normally distributed or not, even with large skewness or nonlinearities in your data. And even if the linear models work *better* on normally distributed data, that does not mean that other models will work better or worse than models that assume normally distributed data. Every problem is different. You will have to develop your own intuition about which models work best on different kinds of data."
-
  Q: "How can I be sure that I will have a complete machine learning pipeline built in the 4 weeks I have remaining for this project?"
  A: "Built a minimum complete pipeline first. Even if your pipeline simply produces random guesses, it will give you the framework to begin improving the pipeline by adding additional features and transformations one at a time. By incrementally improving your pipeline you'll be able to see progress towards your goals if you record your results and hyperparameters in a table that you can review at each step. Plus, you can halt the project at any point and have a complete pipeline whose results will be valuable, if nothing else to show how difficult the problem is."
-
  Q: "How should I process the enron email dataset?"
  A: "You can use the `email` package in python to extract the body, date, from, to, and even fields like `Message-ID`. Each of these can be parsed individually to produce ordinal and categorical variables, like the domain names for the from e-mail addresses. Many of the fields, like the subject and body and even the `Date:` and `From:` can be parsed like semi-strucured or unstructured natural language fields. The more of this metadata you can extract as numerical features, the better your model will perform."
-
  Q: "How do I type the eñe character on a Mac or PC?"
  A: "On a Mac press [option]-[~] then [n]. On a PC type [ctrl]-[~]-[n] and then [n]."
-
  Q: "How do I type the accented e character (as in resumè) on a Mac or PC?"
  A: "On a Mac type [option]-[`] then [e]. On a PC type [ctrl]-[`]-[e] simultaneously."
-
  Q: "I've tried a linear regression, ridge regression, polynomial regression, and random forest model for my home price prediction model. So I have 3 rows in my hyperparameter tuning table. What other models should I try to add more rows?"
  A: "You can vary the hyperparameters for your existing models to get intuition about the affect of those hyperparameters. Example parameters include the alpha parameter for Ridge regression or the polynomial order for your polynomial regression. Check out the `sklearn` documentation and think about which arguments to each of your model's `__init__()` constructor. Think about which ones might affect the accuracy of your model."
  A: "You can also look at your data cleaning steps to see how you might do things differently. This may also affect your accuracy and add to your intuition to help you on future problems."
-
  Q: "Is R^2 a good metric to put in my hyperparameter table?"
  A: "When you present your results what questions do you think your audience will have about the accuracy of your model? Will your audience or boss care about R^2 correlation coefficients? What accuracy number would they need in order to make good business decisions based on your model and the accuracy you are reporting?"
  A_student: "The audience would probably want to know about the maximum dollar error in the price prediction for each home. Sellers could use something like a 60%, 90%, or 99% confidence interval to make business decisions based on how over or undervalued the home listing is relative the that confidence interval."
-
  Q: "My data (word counts) has a really broad range so the histogram looks like an exponential curve and it's hard to see the values for the smaller counts. Should I split it into two plots for the small and large word counts?"
  A: "No. You should use a log scale on the vertical axis of your histogram."
  A_student: "But my audience may not be able to interpret logarithmic scales."
  A_teacher: "Then you may have to educate them. Two separate linear plots with different vertical scales side by side will be misleading. You will be helping them draw incorrect conclusions. You will encounter many logarithmic and exponential datasets in data science. To get the most out of that data you will have to visualize it in a way that is both accurate and informative. Logarithmic scaling is an invaluable tool for that."
-
  Q: "How important is knowing SQL and python programming for a data scientist?"
  Q: "How important is it for a data scientist to know SQL and python for a data scientist?"
  A: "Start with learning python. You can get familiar with data queries and joins using `sqlalchemy`, `Django`, or even Pandas `DataFrame` APIs. These provide a high level abstraction on SQL that builds on your understanding of python. So you can do SQL queries in python without learning another language and syntax. This will free up brain cells to learn more advanced python and data manipulation approaches. Only if you have a performance/scaling challenge (your queries are taking too long) should you dig into the SQL. But you'll find that it takes as much effort as learning python and you will have very limited use for your new SQL skill."
-
  Q: "What activation function should I use for my output fully connected layer (`keras.layers.Dense`) when my target variable is an ordinal. For other neural networks people mostly use `softmax`. My target is adoption rate of pets (0 through 4) in the [\"Shelter Animal Outcomes\" Kaggle competition](https://www.kaggle.com/c/shelter-animal-outcomes/data) dataset."
  A: "You should definitely **not** use softmax for an ordinal target variable. You have a regression problem, not a classification problem. What does a `softmax` function do? Does it have one or many inputs and outputs? It's worth your time to read up on all the activation functions like `sigmoid` and `relu` in addition to `linear`. You can play with them if it's not obvious what they do or whetehr they are appropriate for your problem."
-
  Q: "How can I use `curl` or `wget` to download an image or data file on a web page that has a username and password dialog box?"
  A: "This `curl` command will work on any website that uses a standard HTTP `.htaccess` file for the username and password: `curl -u \"$SITE_USERNAME:$SITE_PASSWORD\" 'http://example.com/path/to/file.csv'`. If you don't want your password stored in your `.bash_history` or shell history, you can use: `curl -u \"$SITE_USERNAME\" 'http://example.com/path/to/file.csv'` and you will then be prompted for the password."
-
  Q: "How can I put my Google Slides on GitHub with my code?"
  A: "You need to download it as a file that others can use, like an `.odp` Open Office Presentation file."
-
  Q: "Does GitHub support slides like `.odp` Libre Office Presentations?"
  A: "Yes, but they aren't editable or viewable on the GitHub website. Like all binary files associated with your code you should edit and view them on your local copy of the `.git` repository."
-
  Q: "Would unsupervised learning for classification of the Enron dataset make a good project to learn Data Science and NLP?"
  A: "Not really. In order to learn from your project you need to know how good your model is at making predictions. Without a labeled dataset, that's really hard to do. A _sillouette score_ can give you a qualitative assessment of the quality of your class label predicitons, but unless you have a business use case for those labels where you can measure the effectiveness of them for that use, you won't have a quantitiative measure of the usefuless of your model. Fortunately Berkeley grad students have categorized some of the enron email dataset into about 30 hierarchical categories or classes: [bailando.sims.berkeley.edu/enron_email.html](http://bailando.sims.berkeley.edu/enron_email.html)"
  Goal: "Supervised learning problems are much better for learning Data Science."
-
  Q: "What happens when you set `sample_weights` equal to your target variable (home price) in a `scikit-learn` model?"
  A: "It multiplies the home price by the loss to emphasize the error for expensive homes and deemphsize the error for lower priced homes during optimization of the weights."
-
  Q: "Why does setting `sample_weights` to your target variable (home price) cause underfitting?"
  A: "Because some of the errors are not given much weight so they don't affect the fit as much and the regression doesn't match the data as closely in those places. It causes underfitting over `all` your data on average, but causes overfitting on the parts of your data where there are high weights (like expensive homes in your example)."
-
  Q: "How can I brush up on my SQL skill?"
  A: "Install sqlite and use python in a jupyter notebook to issue queries on a sqlite database. Any projects that have multiple tables of data with relationships between those tables can benefit from using a sqlite database rather than DataFrames. Try replacing some of your Pandas data cleaning steps with sql queries."
-
  Q: "I'm having trouble understanding the data format in the enron email dataset here: [bailando.sims.berkeley.edu/enron_email.html](http://bailando.sims.berkeley.edu/enron_email.html) text files. What should I do?"
  A: "Read one email and see if you can figure out which of the categories each of the integers refers to. And if you don't have time to develop your data ETL (Export, Transform, Load) skills then you can try this pre-cleaned version of the classified Enron e-mail dataset by [Brian Ray](https://data.world/brianray/enron-email-dataset)."
-
  Q: "My Kaggle dataset has multiple facets. One csv file has pet characteristics like bree, cat/dog, hair length, color. Andother table has metadata about an image of the pet, such as the labels returned by GCV or another computer vision algorithm. Another CSV files had metadata on the natural language description of the pet, such as a TFIDF vector. What can I do to easily incorporate the metadata into my model."
  A: "The image labels can each become a new binary feature in your model. You don't have to incorporate them all. Instead you can just add one at a time and see how it improves your accuracy or starts to increase overfitting (reduced accuracy on your testset)."
-
  Q: "What does the [Slalom](https://www.slalom.com/) company do?"
  A: "[Slalom](https://www.slalom.com/) sends conultants (data engineer, data scientist, and/or business analyst teams) to medium to large corporations to help them figure out what to do with their data. Like if a distributor wants to improve their efficiency base on data about their routes, drivers, and packages. This can often be more of a process optimization problem than a data science problem."
-
  Q: "How do I use a different tokenizer in the sklearn `TFIDFVectorizer`"
  A: "You need to instantiate the TFIDFVectorizer assigning your callable tokenize function to the tokenize argument within the `TFIDFVectorizer` constructor."
-
  Q: "What does it mean when the docs for `TFIDFVectorizer` say that its constructor takes an argument \"tokenize\" that must be a `callable`?"
  A: "That means the argument must be the function name, not the string returned by calling the tokenizer, e.g. use the instance method `tokenize` like this: `tfidfvectorizer = TFIDFVectorizer(tokenize=nltk.Tokenizer().tokenize)` **not** the class method: tfidfvectorizer = TFIDFVectorizer(tokenize=nltk.Tokenizer.tokenize)."
-
  Q: "How can you do dimension reduction on a high dimensional vector like a TFIDF vector or an image?"
  A: "PCA is the most straightforward technique. It is a linear transformation that maximizes the distance between points that were far apart in the high dimesnsional space. TSNE is a nonlinear transformation that mainains the closeness of points that are close together."
-
  Q: "How can I create a document vector using spacy?"
  A: |
    You can do it yourself by summing up all the word vectors:
    ```python
    >>> !python -m spacy download en_core_web_md
    >>> import spacy
    >>> nlp = spacy.load('en_core_web_md')
    >>> docs = ['Hello world!', 'Another doc, another $.', 'Goobye world...']
    >>> pd.DataFrame((pd.DataFrame([w.vector for w in nlp(doc)]).sum(axis=0) for doc in docs))
            0         1         2         3         4         5    ...       294       295       296       297       298       299
    0 -0.019890  0.659450 -0.179160 -0.257430  0.778790 -0.346290  ...  0.677553  0.056050  0.399765  0.059472 -0.580639  0.596957
    1 -0.940003  0.830166 -0.574090  0.328137  1.504906  0.614334  ... -1.912200  0.324076 -0.189071 -0.382730 -1.281060 -0.331810
    2 -0.004859  0.471100  0.191603 -0.432700  0.321437 -0.309723  ...  0.254403  0.313342  0.050089  0.073897  0.047953  0.240027
    ```
    Or you can just use the doc vector computed internally by Spacy the exact same way (except Spacy calculates the `.mean()` rather than the `.sum()`:
    ```python
    >>> pd.DataFrame((nlp(doc).vector for doc in docs))
            0         1         2         3         4         5    ...       294       295       296       297       298       299
    0 -0.006630  0.219817 -0.059720 -0.085810  0.259597 -0.115430  ...  0.225851  0.018683  0.133255  0.019824 -0.193546  0.198986
    1 -0.156667  0.138361 -0.095682  0.054689  0.250818  0.102389  ... -0.318700  0.054013 -0.031512 -0.063788 -0.213510 -0.055302
    2 -0.001620  0.157033  0.063868 -0.144233  0.107146 -0.103241  ...  0.084801  0.104447  0.016696  0.024632  0.015984  0.080009
    [3 rows x 300 columns]
    ```
    Notice that the first document vector computed by spacy is 1/3 the magnitude in each dimension than the sums that we computed for those dimensions, because "Hello world!" tokenizes into 3 tokens 'Hello', 'world', and '!'.

    Also, notice that mispelled words that don't exist in the spacy dictionary, like "Goobye" don't crash the loop. Spacy just returns an all-zero vector for those words.

    See the [spaCy documentation](https://spacy.io/usage/vectors-similarity) for more details.
-
  Q: "What are the different kinds of data science or machine learning features or predictor variables?"
  A: "There are really on two kinds of features that can flow directly into a machine learning or data science model Continuous and Binary. In fact even binary features are treated as continuous features. All other data science features need to be converted into these types before it is ready for modeling with statistical or machine learning models. For example, categorical variables must be converted to an appropriate number of binary variables. Ordinal variables can be treated as categorical or continuous, but it's generally best to use them directly as continuous values once they are converted to numerical values with the appropriate order. Natural Language and Images are often converted to highdimensional binary or continuous vectors. Date, Time, and DateTime objects are usually converted to continuous values by subtracting two dates or times. They can also be converted to categorical or binary variables like Sunday, January, Summer, Rush Hour, Business Hours, Night. etc. Similarly latitiude and longitude and other location information can be converted to continuous values like distance from a landmark or categorical variables such as neighborhoods or addresses or states/counties/zips."
-
  Q: "Should you use a dictionary or a list to store a mapping between integer user IDs and usernames?"
  A: "If the integers are consecutive and start at zero and you want to persist them perpetually (not ever delete an ID->user mapping), then a list is most efficient. If there are gaps or the integers do not start at 0 or 1 then you should use a dictionary."
-
  Q: "What does \"naive\" mean in a \"Naive Bayes\" model?"
  A: "It means that all your features are assumed to be conditionally independent form each other. It's typically used for natural language word frequencies as your features. Each record is a bag of words vector. So you're assuming that the occurrence of one word in one document does not affect the probability of another particular word occurring in that document.  This is not a very accurate assumption. But it worked well enough to save us from the onslaught of spam in the early 00's."
-
  Q: "When I did a `.value_counts()` and then a `df.plot()` but it tells me that the dataframe is empty because it doesn't contain any numeric data. What gives?"
  A: "Examine the dataframe in your variable df to see if it contains any numerical data for the columns you've selected. You may have done your `.value_counts()` and not saved the Series that `.value_counts()` returns into a variable. So you're just calling `.plot()` on the original data with uncounted categorical data (strings) which can't be plotted."
-
  Q: "What is the difference between a null value in pandas (`df.isnull()`) and a nan value (`df.isna()`)?"
  A: Null values include both `np.nan` and empty strings like ''.
-
  Q: "What is the difference between a numpy `array` and a pandas `Series`?"
  A: "A Pandas series is indexed according to a label, similar to how a dictionary is indexed by keys. A numpy array is only indexed by the consecutive integers of the positions of the objects within the array. When you perform mathematical operations, array elements are operated on in the order they appear within the array. When you performan mathematical operations on Series only matching indices in a binary operation are operated on together. "
-
  Q:  "What data science packages do you use regularly in addition to pandas, numpy, scikit-learn, jupyter, matplotlib?"
  A: "Django or sqlalchemy for database ORMs. Spacy for NLP. OpenCV and Keras for computer vision (images). Plotly for interactive visualizations and dashboards."
-
  Q: "How can I plot red dots for one crime type and blue dots for another crime type on the same plot?"
  A: |
    You can specify a different color for two different plots on the same `axis` if you pass your new axis into the next plot method call.
    ```python
    ax = df[df['crime_type']=='crime_type1']].plot(kind='scatter', x='lon', y='lat', color='r')
    ax = df[df['crime_type']=='crime_type2']].plot(ax=ax, kind='scatter', x='lon', y='lat', color='b', alpha=.1)
    ```
-
  Q: "My model gets 99% accuracy on the fastball no-fastbball prediction problem."
  A: "You are probably \"cheating\" by allowing some leakage from your target variable into your predictor variables."
-
  Q: "In `sklearn.train_test_split` does a value of test_size=.25 mean that 25% of the data will be reserved for training or for testing?"
  A: "As long as the return arguments are in the order `x_train, y_train, x_test, y_test` then the test_size argument controls the size of the samples in the test_set (`x_test` and `y_test`). But as with all data transformations you should verify that your intended transformation has been accomplished. In this case this means verifying the `.shape` of the 4 returned variables after having run `train_test_split()`."
