-
  Q: "Is it possible to select a subset of rows and columns based on conditional expressions like I do in R?"
  A: "Yes! For example, this would select the columns named 'aaa' and 'bbb' and then filter them to only return the rows where the column named 'zzz' has values greater than 0.5: `df[['aaa', 'bbb']][df['zzz'] > .5]`"

-
  S: What is the difference between a Data Analyst role and Data Scientist role?
  T: What is Analysis? And what is Science? How are they different?
  S: Analysis is a lower level job. Data Scientist is a level above that. And Machine Learning is a level above that.
  T: Yes, but what is different about the actual work of Science vs Analysis?
  S: Analysts look at data with visualizations and statistics. They provide insights to management. Science is more structured, has a whole process, the scientific method.
  T: Exactly. A Data Scientist will typically work with the same data as an analyst, but instead of just hand waving and suggesting *ideas* for what the business should do, a Data Scientist provides a *predictive model* of the world. And that model's correctness can be measured quantitatively. Science doesn't depend on human intuition, and isn't distracted by human biases or logical fallacies. Science let's you measure exactly how correct your model of the world is.
-
  S: Am I selling myself short if I apply for a Data Analyst role rather than Data Scientist?
  T: Not at all. Your goal is to get a job where they pay you to look at data and learn about how to work with data. If they ask you to analyze a dataset, you can use your Data Science and Machine Learning models to blow their socks off with "analysis" that gives them actionable insights. And you can give them confidence intervals on the range of possible outcomes that will result from those actions. At the very least you can provide them with predictions that help them anticipate what they need to do to improve their business.
-
  S: I'm playing with a GPT-J demonstration and using it to generate answers to provocative, controversial questions. I'm trying to see how prosocial or socially responsible its dialog is. Won't its responses just reflect the biases in the data it was trained on? Garbage in, garbage out?
  T: Yes indeed. What do you think you can do about that?
  S: Can I train it on data that reflects my values so that it answers questions prosocially?
  T: You could, but that would require a lot of data. Alternatively you could fine tune the model. That would require much less data. But that would create a language model that doesn't even know what some offensive words and concepts are. It would not understand the grammar and tone of that offensive dialog that you excluded from the dataset. The ability of an artifical brain to talk in offensive antisocial ways is not a bad thing. It's a good thing. It represents accuracy at generating human-like natural language text. A machine learning model like GPT-J doesn't exist in isolation. Can you think of a way to make a GPT-J-based chatbot more prosocial, without retraining or fine tuning?
  S: Can I make sure the prompts I send it are prosocial, so that I keep it on track?
  T: That's a great idea. That would mean changing the input to the dialog to paraphrase what someone says to the chatbot in a way that is more prosocial and mindful and thoughtful. To do that you will need to have a measure of the prosocialness score that you are trying to optimize for. You need to know when what you are doing is quantitatively improving what the bot says and when it is not. Can you think of a way to modify the output of GPT-J rather than the input or training data?
  S: Ahh, yes. I could modify the output just like I modified the prompts or questions from the human.
  T: Right. And you may not be aware that you can also ask GPT-J to generate many alternative responses and you can select from among them the one that is closer to what you would like it to say. Once you have a way to measure, or score, sentences fro prosocialness, then you can use that to rank the alternatives and chose the one that reflects your values.
  S: Oh I didn't know that. You can ask for a do-over from a transformer model.
  T: Right. And you can adjust the "temperature" of most models, so that the variance or randomness of the text it generates increases.
-
  S: What can I do to create a sentiment or intent recognition classifier to score or classify natural language text as prosocial or not?
  T: You can use the embedding or encoding from a transformer model such as BERT or GPT-J as the input feature vector for each text you want to classify. Then 
  
-
  Q: What is a good package for doing topic analysis or natural language analytics?
  A: The ldaviz package has good visualizations in addition to the Latent Dirichlet Allocation topic vector analysis.
-
  Q: What is dbscann?
  A: Unsupervised clustering algorithm that choses a random point and gathers up all the points within a radius into a cluster. No need to specify the number of clusters, just 2 radiuses used to grow and stop growing the clusters.
-
  Q: When should I start networking for my career search.  #career
  A: Yesterday ;)  Join meetup groups, Slack workspaces, Discord channels, and LinkedIn groups that interest you. Follow your curiosity, and share that curiosity with others. Not only is that the fastest way to find work that you enjoy, but also to discover friends and ideas that inspire you.  And don't let geography limit your networking. Start where you are, but work your way out to all the hubs of technology and data science and AI, like [Torronto Machine Learning Meetup](proai.org/tmls) [San Diego Python](proai.org/sdpython) [PDX Python](proai.org/pdxpython), [SDML](proai.org/sdml-slack), etc
-
  Q: I found a blog post about turning images into line drawings. I'd like to create a model to create vector/line drawings from images.
  A: Not a good idea. That is not in the form of a conventional supervised learning data science problem. Your first data science project should be a supervised regression or classification problem that has both conventional features (numerical, categorical) as well as one or two special features (datetime, natural language, location, image, audio)
-
  Q: What should I do to get started on a real data science problem?
  A: |+
      1. find a dataset with at least 100 rows
      2. chose a target variable
      3. tell me what kind of model it is
      4. chose a feature variable
      5. tell me what data cleaning you need to do on the feature variables and target variables
      6. train a model on the entire dataset
      7. predict the target variable for the first row in your feature vector data frame (`X.head(5)` or `X[:5,:]`)
      8. look at the target labels for those first 5 examples (y.head(5) or `y[:5]`) to see how good or bad your model's predictions are 
      9. imagine an example feature vector (if you only have one feature it will be a 1-D vector) and create it in code
      10. predict your target variable for that example you created in 9
      11. how accurate do you think your model will be in the real world?
-
  Q: How can an object detector be used to build a 3D position and 3D pose detection algorithm in a robotic environment with a pick-and-place SCARA robot end-effector attempting to put tools or appliances or cuvettes into the appropriate "nest."
  A:
      - Geoffry Hinton's new "capsule networks" may be useful. They output at 2-D pose/orientation representation of the detected objects, in addition to bounding box information. So first step would be to reproduce his results on the multi-mnist dataset.
      - You can use a Kalman filter to track the relative position and orientation of the end-effector camera relative to a fixed target object in the environment. And the negative (inverse) of this vector represents the relative orientation of the target. And the kinematics of the robot can be used to estimate the inertial-space position/pose of the target object.
-
  Q: How can I compete with big banks in time series forecasting of index funds?  #project #capstone #finance #time_series #forecasting
  A: 
    - Pay attention to the social impact of your investments. Big banks are run by antisocial people that don't care about  the impact of global warming and social-responsibility. And they don't realize the magnitude of its affect on the markets and their investments.
    - Focus your model on predicting risk-adjusted long term performance of your investments. Make your target variable the value of assets at least 2 years into the future. Most investment bankers and funds focus on the near-term performance of their assets and rely on active trading to manage long term performance. So your model will be better than theirs for predicting long term performance and it will minimize the amount of trading fees you have to pay.
- 
  Q: Name at least 8 kinds of data used in data science as features in a machine learning problem. Imagine a housing price dataset, or an product shopping cart dataset. What are all the different kinds of data objects you would see in a shopping cart form or in a house listing. And list them from easy to hard, in terms of their dimensionality or the complexity of the data wrangling and feature extraction required.
  A: |+
   Examples in parentheses for a housing dataset: 
    1) continuous numerical (price, sq ft)
    2) categorical (paint color, duplex or not)
    3) ordinal (single/double/queen/king bed)
    4) natural language (description)
    5) datetime (date of sale)
    6) location (zip, lat/lon)
    7) audio (audio clip of street noise)
    8) images (aereal images)
    9) videos (home walkthrough)
    10) spatio-temporal data (city crime, weather)
-
  Q: What are some time management tricks I could use to improve my productivity?
  A: Have you ever tried a "power hour" with a colleague or a friend? Set a regular time to meet with someone on Zoom or video conference. At the beginning of the meeting you just say hello and maybe explain what you hope to accomplish in the hour. The other person does the same. Then you just put yourselves on mute. Having the video of someone watching you can help keep you focused, due to some natural instincts that all social mammals share. We care what other people (and even animals) think of us.
-
  Q: Are there any tricks for helping me get good (more efficient, more productive) at writing software?
  A: Every time you solve a bug or succeed quickly at some programming task, reflect on it and think about what worked. Did you use Duck.com for a particular word or phrase from an error message? Did you dive right in and start coding a solution or did you pause to think and write a documentation string before writing your first line of code? Whatever it was, write it down. Take note of what works (and what doesn't). Just the act of writing it down will make you more mindful of th egood and bad decisions you make during the day. 
-
  Q: What are some time management tricks I could use to improve my productivity?
  A: Have you ever tried a "power hour" with a colleague or a friend? Set a regular time to meet with someone on Zoom or video conference. At the beginning of the meeting you just say hello and maybe explain what you hope to accomplish in the hour. The other person does the same. Then you just put yourselves on mute. Having the video of someone watching you can help keep you focused, due to some natural instincts that all social mammals share. We care what other people (and even animals) think of us.
-
  Q: Should I work for a large company or a startup for my first job? Isn't it a bad thing if the company goes bankrupt after a few months on the job?
  A: I like working for startups. You get to work with smart people and learn all different aspects of the business. Follow your curiosity. Read job descriptions and look for ones that interest you and find out what skills are popular for jobs like that. Build up your skills and professional network to make yourself valuable to those companies that interest you.
-
  Q: |
    This is the command to train the model, which the authors have provided in their github -- 
    `python train.py -task abs -mode train -bert_data_path BERT_DATA_PATH -dec_dropout 0.2 -model_path MODEL_PATH -sep_optim true -lr_bert 0.002 -lr_dec 0.2 -save_checkpoint_steps 2000 -batch_size 140 -train_steps 200000 -report_every 50 -accum_count 5 -use_bert_emb true -use_interval true -warmup_steps_bert 20000 -warmup_steps_dec 10000 -max_pos 512 -visible_gpus 0,1,2,3  -log_file ../logs/abs_bert_cnndm`.
    If you observe, the train_steps is 200000, which is a huge number. Should we proceed with the default values that they have given or reduce the train_steps? I am not sure how much impact it's gonna have on the training time and accuracy.
  A: Try to estimate the runtime to answer your own question. If you haven't already, run the training on a smaller number of samples from your dataset and a smaller number of training steps (epochs) to help you compute this estimate.
-
  Q: Although the code enables the distributed training if it detects multiple gpus, Is there any way to verify manually if we are utilizing all the GPU's while training?
  A: "Ask that question on DuckDuckGo.com. (spoiler alert: `nvidia-smi`)"
-
  Q: Since, training takes a lot of time, even on the GPU server, If I  move away from my laptop and if it enters into sleep mode, will the training still continues or it halts? Because, in Google colab, if the user is idle for some amount of time training stops.
  A: "Good question. any time you run a command on a remote server you need to be able to disconnect from the ssh session and have the process continue to run in the background. Duck.com for \"tmux\" and \"ssh connection disconnects server\""
-
  Q: Is there any way to start the training and return back to it once it finishes, without being worried about the idle timeout, or sleep issues, which I mentioned in the previous question about disconnecting from the server when my laptop sleeps?
  A: The `tmux` application will solve this challenge. Also the script should be saving checkpoints in the background, so you can always halt the process (ctrl-C) and you will still have the last checkpoints that you can load in order to restart the training from that point.
-
  Q: I have a time series of assignments in a 10 week self-paced online college course. I have data for several STEM and non-STEM courses. I'd like to predict student's passing the course.
  A: First process the raw data to create a binary feature for completed an assignment. Discretize the data into days, with one day for each day in the course. All dates are relative to the beginning of that particular course for that particular student. Then compute a rolling window mean of the binary variable 14 days in the future. This can become your target variable. The feature variable is the actual zeros and ones for that day. You can have a column for the zeros and ones for the assignments completed today. Another column for zeros and ones for the assignment completed yesterday, and so on, going back in time as far as you like. So if you went back in time 7 days for your features your would have 7 binary columns for features. And you would have one target variable for the 14 day mean in the future.
-
  Q: My jupyter kernel keeps crashing. 
  A: You need to monitor you memory usage and make sure it isn't swapping.
-
  Q: Is swap when you are creating variables?
  A: No. Swap is when you overflow the RAM (volatile memory) and it has to create variables and objects on disk (persistent memory). Disk writing and reading is 100x or 1000x slower than reading and writing from RAM. And if you CPU gets bogged down with multiple processes reading and writing from disk it can freeze or crash the kernel.
-
  Q: How can I reduce my memory (RAM) usage?
  A: "First you have to find out which lines of code are using the most memory. For that you can run `htop` in a terminal window or tab to the side of your screen. on mac: `brew install htop`. On linux `sudo -apt-get install htop`"
-
  Q: How can I monitor my memory (RAM) usage?
  A: "Run `htop` in a terminal window or tab to the side of your screen as you run your code a few lines at a time. If you need more detailed information about which functions and "
-
  Q: Which lines of python code are using up the most RAM?
  A: Focus your attention on objects that contain training or testsets for machine learning, especially processed images or natural language. Use an external tool like `htop` to monitor CPU and memory usage. Some python tools and packages can also measure memory usage. The builtin `sys` package contains the `getsizeof()` function which can tell you the approximate number of bytes used by any python object. Also, this [pluralsight tutorial](https://www.pluralsight.com/blog/tutorials/how-to-profile-memory-usage-in-python) suggests a tools called `hpy` within guppy (pip install guppy). It displays all the objects in your heap and their memory usage. 
-
  Q: In a job interview for a data science position they are asking me business questions about how I can determine the objective or problem statement for a data science project.
  A: Explain to the itnerviewer that you would need to interview the business stakeholders (managers) to find out what their objectives are and see if there are ways to measure these variables in existing datasets that are collected regularly. Unfortunately, that will usually bias the model towards short term business success (profit and revenue), because that's what most business managers care about the most. It can help to try to build a predict variables like profit and revenue a year in advance. This will ensure you are optimizing for long term business success. It would be even better if you could interview users (the forgotten stakeholder) rather than business managers. You want to find out how they interact with your system when they are extremely satisfied or dissatisfied with your product. You will also need to watch out for confounding variables or spurious correlations. You can check for the dosage effect -- the error (residuals) should be uniformly distributed around your predictions, irrespective of the magnitude of the predicted variable. The accuracy or power of the machine learning model for the predictive variables should be high. Using Lasso or Ridge or other machine learning model regularization techniques can help ensure this. Normalizing for as many variables as possible by including them as machine learning features in your training dataset can help. Engineering many features based on the residual plots can also help ensure that confounding variables have been normalized away. Also, there needs to be a theoretical or physical reason for why the business managers sustpect that the dominant feature variables could cause the target variable to change. In the end, the only way to be sure the effect is causal, is to perform an experiment, like an A/B test or a randomized controled trial (preferably double-blind). 
