-
  Q: "Why did my statsmodel OLS (Ordinary Least Squares) model get better accuracy without the constant term (bias or intercept)?"
  A: "You must have made a mistake. Adding a feature or intercept term or additional degree of freedom to a model almost always improves accuracy. Perhaps you are not adding the constant to the correct object in python? Make sure you run an entire jupyter notebook top to bottom whenever you make significant changes."
-
  Q: "Can you recommend any resources for learning Python?"
  A: "We use this list of resources for our interns. It's under the reources tab at TangibleAI.com: https://tangibleai.com/getting-started-with-python-and-data-analysis/"
-
  Q: "How can I create a statistical model to predict diabetes from age if I have a table of patient ages and a binary categorical variable for whether they have diabetes?"
  A: "The most basic model you can create is to predict the probability of someone having diabetes in your entire population by taking the mean of the binary categorical variable. A better model would divide the age into categegories, say  teenager, adult, and senior citizen. And then calculate the mean of your diabetes column for each of those age categories."
-
  Q: "I want to create a weighted harmonic mean of precision and recall as my performance metric on my project."
  A: "It would be much easier to explain your results to other data scientists if you used a more standard score like F1-score."
-
  Q: "What is a good project for Data Science in Healthcare?"
  A: "There are public datasets on county-level health statistics like diabetes and cancer. These datasets can be used to create interesting visualizations and predictive models based on geographic and demographic data."
-
  Q: "How can I create a test set for a recommendation engine trained on individual movie likes by users?"
  A: "Segment your data according to user id and treat your entire test set like you would treat a new user on your system."
-
  Q: "I have a log file of 911 calls including addresses and text transcripts of the incidents that I'd like to do Data Science on to see if covid 19 has has caused an increase in suicide attempts."
  A: "That's a potential privacy violation by the Orlando Police Department. Be careful with that data. You will want to join it with a table of features associated with COVID19."
-
  Q: What do the weights (.coef) of an `sklearn.LinearRegression` represent for my model that predicts Air BnB price based on features like number of reviews?
  A: For the particular coefficient that you mentioned that would represent the change in price or value that your stakeholder could expect for each review they were able to add for a particular property.
-
  Q: I have plotted the weekly death rate for cancer for each year as a separate series (ax) on a Matplotlib plot. I'd like to label the x axis with a month/day for each week, rather than a week number.
  A: The problem is, the week number and the date are different for each year, so you would have to carefully resample each year with a weekly date to get usable dates that are consistent for each year. Instead it might be helpful to just label the months approximately using the xlabel or xtick arguments to a matplotlib plot.
-
  Q: "I'm getting 90% accuracy on my test set from one set of fake news and authentic news articles. But when I try the Kaggle dataset I find that it does poorly because the Kaggle dataset contains a lot of Russian, Arabic, and other foreign text and non-ascii characters."
  A: "First write up your draft report, including a hyperparameter table for the models and hyperparameters you've already tried. Then think about all the approaches you can use to generalize a natural language processign model. One of the best ways is to reduce your vocabulary with `min_df` and `max_df`. Long term you can consider using pretrained language models like SpaCy or BERT."
-
  Q: "What is Jaccard similarity?"
  A: "Jaccard similarity is also called Intersection Over Union (IOU). It is a measure of the similarity between two sets. It is computed as the length (number of elements) in the intersection of the two sets divided by the length (number of elements) of the union of the two datasets."
-
  Q: I get 0.1% R^2 score on my training set for a LinearRegression with death rate as my target or response variable (y) and zip code as my only feature variable. What's wrong?
  A: What kind of variable is zip code?  Categorical or numerical?  What do you need to do to all categorical variables before passing into a LinearRegression?
-
  Q: I have a rust application I'd like to run. Even though the bin directory where it is located is in my PATH, bash console can't find it an run it as a command without the full path specified.
  A: That's either because the permissions aren't executable and readable by the user you're running it as or your using sudo on a program located in the root user's directory rather than your own user's directory.
-
  Q: How can I do hyperparameter tuning on multiple hyperparameters in a loop?
  A: You will need to create nested loops that iterate through each of the lists of hyperparameters and assign them to the appropriate command line arguments to your model or pipeline within the innermost loop. You entire model fit, predict, evaluate steps should be in that inner loop and recorded in a list of dictionaries or concatenated to a dataframe.
-
  Q: How can I deal with outliers in my target variable?
  A: You may want to create more categorical or nonlinear features from your features. Look for discontinuities in your residual plots and create thresholds for binary features around those discontinuities. Also transforming many of your features with log or exp or x**2 might also help.
-
  Q: My training runs take forever.
  A: Make sure you monitor your RAM usage during the run. If you see the RAM usage constantly growing that means you have a "memory leak". And if the RAM usage ever exceeds what your machine can handle, the training process will slow to a crawl due to what's called "swapping".
-
  Q: Do you think an NLP problem would be too hard for me?
  A: It depends on the dataset. I think you can handle any dataset that includes a variety of features, like continuous variables, categorical variables, time or location variables, even if it has some natural language text data. You can start simple by cleaning and modeling the easier variables first and then work your way up to more and more complicated natural language processing techniques.
-
  Q: I can't find my new file on gitlab.com, even after I `git commit` and `git push`.
  A: You probably haven't added the file yet. I always use `git status` before doing anything within a git repository. It will highlight in red any files that need to be added. I also always use the `git commit -am "message goes here"` for my commits. That ensures that any modified files are also automatically added.
-
  Q: What makes machine translation difficult or easy as a data science problem.
  A: Machine translation requires a large dataset of paired sentences, but fortunately for many languages these datasets are publicly available. Another thing that makes machine translation difficult is that it is a generative process. You must train a generative model. A generative model trains one word in a sequence based on the other words in the sequence and the errors for each word compound on each other. The model must learn the grammar and meaning of the languages you are translating from and to. Deep learning is the most successful approach for these complex language models.
-
  Q: "I've created function for use in my hyperparameter tuning loop: `def metrics(model, features, X_test, X_train, y_test, y_train):`."
  A: "Make sure you use named arguments in the function definition as well as the function calls within your hyperparameter tuning loop: `def metrics(model=None, features=None,...):`."
-
  Q: "When I upload my jupyter notebook to github it displays the output, including large data tables that clutter up the document and make it impossible to read and review."
  A: "You can make your jupyter notebooks more presentable by ensuring that any table rows that are displayed in your output cells are relevant and useful. Use `.head()` and `.tail()` and `.describe()`. Don't ever display entire raw DataFrames unless they are small."
-
  Q: "What does `np.array([True, True, False]).mean()` return ?"
  A: "0.6666..."
-
  Q: "I have a rust `exa` application I'd like to run and have it replace my `ls` command. Even though the bin directory where it is located is in my PATH, bash console can't find it an run it as a command without the full path specified."
  A: "That's either because the permissions aren't executable and readable by the user you're running it as or your using sudo on a program located in the root user's directory rather than your own user's directory."
-
  Q: "Should I work on a neural network problem?"
  A: "Not until you've mastered all the other elements of the data science pipeline and worked with all the different kinds of data that you might do deep learning on, like natural language, time series, geographic data, recommendation engine data, and high-dimensional data."
-
  Q: "What feature engineering can I do with 911 call response data?"
  A: "For any GIS data, one of the most important features you can engineer is a distance from a arbitrary landmark. Pick some arbitrary latitude and longitude locations near the region of your dataset and see if the distances from those landmarks helps improve your accuracy."
-
  Q: "At what stage of my pipeline should I do the train_test_split and the scaling?"
  A: "You should fit the scaler after the train_test_split, fitting only to the training set. No `.fit` function should ever be run on a test set or validation set or holdout set. You sohould only ever use `scaler.transform` on the validation set, test set, or holdout set."
