-
  Q: "What are your favorite methods for exploring a Pandas `DataFrame`?"
  A: "Assuming you've loaded your data into a DataFrame called `df`, some good ones are: `df.info()`, `df.describe()`, `df.min()`, `df.max()`, `df['column'].value_counts()`, `df['column'].count()`, `len(df['column'].unique())`, `df.corrwith('target_column')`"
-
  Q: Are there more jobs for data analysts with SQL skills or general Data Science and Machine Learning engineers with Python skill?
  A: Data Science positions are generally considered to be higher skilled jobs. Many data analyst positions are filled by Data Scientists who use their Python skills to produce analytics visualizations and even dashboards.
-
  Q: What does the `==` binary operator return when one side of the expression is numpy array and the other side is a scalar (like a `float`, `int`, or `str`)?
  A: An array of `bool` objects, one for each element in the array, and `True`s corresponding to the presence of the scalar value in the array at that location in the array.
-
  Q: How can I combine numerical features with TFIDF features for a natural language column in my tabular data?
  A: You need to concatenate the dataframe containing your TFIDF vectors and the dataframe containing your numerical features into a single dataframe.
-
  Q: What are some good datasets for training a machine translation pipeline?
  A: Look for wikipedia.org enclopedias in the languages you want to transltate. YOu can train language models in each language based on those two corpora, independently. Next you need pairs of sentences in each language. Here's a [place to start](https://lionbridge.ai/datasets/25-best-parallel-text-datasets-for-machine-translation-training/). And [here's a tutorial](https://github.com/geyang/deep_machine_translation).
-
  Q: What is Ocam's razor?
  A: The idea that the simplest explanation is usually the correct one.
-
  Q: What do I need to measure training set accuracy for if I already calculate test set accuracy?
  A: So you can see if your model is overfitting and use regularization hyperparameters appropriately.
-
  Q: My OLS linear regression model gives higher accuracy on the test set than the Ridge or SGDRegressor.
  A: If you've tuned the Ridge hyperparameters well, it should give you the same or better accuracy as OLS. The same goes for SGDRegressor, but SGDREgressor is much harder to tune because it has so many more hyperparameters to adjust.
-
  Q: What makes a good data science for learning?
  A: A "tall" tabular dataset with 10x or 100x more rows than columns.  It should also have a variety of variable types in the columns. It should have at least a few categorical and numerical feature variables. And it should have a couple special variables rich with data requiring feature engineering. Some examples of rich "special" data types are date, time, location (such as latitude/longitude) or natural language (like a description column).
-
  Q: Where can I learn about mechanistic or deterministic models and how they compare to data science models?
  A: http://www.thetalkingmachines.com/episodes/talking-machines-live-and-understanding-modeling-viruses
-
  Q: "Why isn't machine learning used to forecast the spread of covid-19?"
  A: "This [talking machines episode on epidemiology in Africa](http://www.thetalkingmachines.com/episodes/talking-machines-live-and-understanding-modeling-viruses) explains it well. This [deep learning model to predict covid-19 infections](https://colab.research.google.com/drive/1tsObYOlqiR5yLStNu0XmAOGL6P_KLb7b?usp=sharing) is another example from the other perspective (data science)."
-
  Q: "What is the SQL query to retrieve a list of member names from a table called facilities where the member cost is between 0 and 20% of the maintenance cost for a facility?"
  A: "Don't forget to specify the full scope (name space) of all variables: `SELECT Facility.member_name FROM Facility WHERE Facility.member_cost > 0 AND Facility.member_cost < 0.2 * Facility.maintenance_cost`"
-
  Q: "What is a good book for learning about the philosophy of consciousness, especially for AI?"
  A: "The most impactful book on this topic for Replika CEO, Eugenia Budya, was  _GEB_ or [_Gödel, Escher, Bach_](https://en.m.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach) by Douglas Hofstadter. In this book, programming challenges like like Quine (code that writes itself) and MIU (a recurrent grammar puzzle) were first introduced."
-
  Q: What are some good variables to plot on the horizontal axis of a residual plot?
  A: Any continuous feature variable is best because you can compensate for any pattern you see in the residual plot by using feature transformations. If you see curvature your can add a column that is the square (`**2`) or `sqrt()` of that feature variable.  That might reduce the residual and increase your model accuracy.
-
  Q: Why am I not getting better train test scores when I scale the feature variables?
  A: Scaling rarely helps a model perform better. This is a key intuition you should gain from this course. Also, you are not performing the scaling correctly. Think of the scaler as part of the same pipeline that the model is part of, it must be fit to the training set and the same transformer (that was fit to the training set) should be used to transform both the training set and the test set. No model, not even a scaler, should be fit to the test set.
-
  Q: What are the metrics I need to use for my regression models/ classification models?
  A: The correlation score is OK, but a better one is RMSE, this is called standard error. This is what a linear regression is attempting to minimize.
-
  Q: What other models do I need to use to make the project better?
  A: Check out the Choosing the Right Estimator (model) flow chart in the scikit learn online documentation. As you develop as a data scientist, the decisions in the flow chart will eventually be intuitive to you. You will understand the reason behind each decision so you can make those decisions on your own. Often you will have gone down many of these paths yourself and discovered new models that worked even better than the ones recommended by `sklearn` It's not the model nor the scaler that are the most important decisions that a Data Scientist makes. Rather it is feature engineering. Your problem is a very hard problem, so it will take a lot of creative feature engineering to find those features that are predictive of your target variable.
-
  Q: I used 5-fold cross validation to estimate my model accuracy in the real world, but how do I know which error is the right estimate?
  A: "Why did you do 5 fold cross validation? Did the results tell you anything new about your model? What does 5-fold cross validation do? If you wrote code to implement `kfolds_cross_validation()` what would your code look like? What does `train_test_split()` do? If you wrote code to do `train_test_split()`, what would it look like? Aren't `train_test_split` and `kfold_cross_validation()` redundant? Did you do 5-fold cross validation on your first capstone project? Can you think of a better approach or workflow? Hint: My preferred approach takes 5x less time than 5-fold cross validation."
-
  Q: How can I add a directory path within Juptyter notebook.
  A: In the home page view of Jupyter Notebook you can navigate to any directory (folder). The new button in the upper right allows you to create folders. Then when you click the check box next to an Untitled folder, a "Rename" button will appear in the upper left.
-
  Q: What can I do about negative correlation score for my test set?
  A: You may need to do manual 10-fold cross validation on a smaller train_test_split test_size (10% or 20%). Also try training the model on you single most powerful feature, or turning up the Lasso regularization strength a large amount.
-
  Q: How do you use a trained linear regression model to select only the 10 most powerful features in an ML model?
  A: X[list(pd.Series(model.coef_.abs(), index=X.columns).sort_values().index.values[-10:])]
-
  Q: I am going to be looking deeper into the visual results of that for my next step but also wanted to check in with you and try to get a better idea of how to capitalize on other categorical variables. Specifically, I am trying to figure out how I can better represent the potential of a relationship between variables and how to correctly pair plot.
  A: Try plotting 2 histograms on the same figure with a transparency (alpha) value of about 50% (.5). The value of the histogram can be your target numerical value for your regressor. The 2 histograms can be any pair ov values for the binary dummy variable created from a categorical feature variable.
-
  Q: How do I create a dataframe with column names for each of the features from an `sklearn.featureextraction.PolynomialTransformer()`?
  A: Review this [stack overflow answer](https://stackoverflow.com/a/36735970/623735) which suggests `column_names = ['x'.join(['{}^{}'.format(k,k) for (k,v) in names if v]) for names in [zip(df.columns,p) for p in xformer.powers_]]`
-
  Q: The way I had the models set up to run, was to run them at 100, 500, and 1000 trees, with 36 models (12 separate cell blocks), to test my selected hyperparameters, but upon trying the base Random Tree Regression model (no hyperparamter tuning, and running it at 100, 500, 1000 trees), the model went over 12 hours with no signs of stopping. To adjust for this, I tried tuning down the trees to 100, 300, and 500 on the base model, but it ran over 12 hours with no results (I stopped it at that point).
  A: When ever running a long-running process on a computer you **must** break it into as small pieces as possible and run those pieces manually, one at a time. And with each of them, try to use something like `tqdm` to estimate the amount of processing required. Also. On high dimensional datasets (or any dataset, actually) you should always train the simplest possible model first. What is the simplest model in the sklean package for your problem?  What kinds of models did the curriculum recommend for high dimensional datasets like NLP vectors? And what hyperparameters would make a Random Forest even simpler than the ones you suggested?
-
  Q: What is the first model I should use to predict a continuous numerical value?
  A: Always start with the simplest, easiest to explain, and fastest model to train. Usually this is a Linear Regression for a regression problem (continuous numerical target variable).
-
  Q: What is a good workflow for finding datasets and planning a project?
  A: 1) duck.com search for "download machine learning dataset <intersting industry or data type keywords here>". 2) Browse to those links that look intersting. 3) download any CSVs you find 4) load csvs into pandas and examine the `.shape` and `.head()` for each. 4) make sure they have an interesting target feature column and a tall dataset (more rows the columns)

