# Key Prompts for the spark-expectations Repository

This document provides over 50 important prompts to help you explore, configure, develop, and troubleshoot the spark-expectations project. Use these prompts with GitHub Copilot Chat or as a checklist for your work.

---

## Getting Started & Setup
1. How do I set up the development environment?
2. How do I install dependencies for this project?
3. How do I run the sample notebooks?
4. How do I start the local SparkExpectations environment with Docker?
5. How do I generate mail server certificates for local testing?
6. How do I build the documentation?
7. How do I remove all hatch environments?
8. How do I upgrade project dependencies?
9. How do I install Spark-Expectations from a specific git branch?
10. How do I install Spark-Expectations from PyPI?

## Data Quality Rules & Configuration
11. How do I create the rules table for data quality checks?
12. What columns are required in the rules table?
13. What are the valid values for the `rule_type` column?
14. How do I add constraints to the rules table?
15. How do I define a row-level data quality rule?
16. How do I define an aggregation data quality rule?
17. How do I define a query data quality rule?
18. What does the `action_if_failed` column do?
19. How do I set up error drop alerts for rules?
20. How do I set rule priorities (low, medium, high)?
21. How do I enable or disable specific rules?
22. How do I use tags and descriptions for rules?
23. How do I enable source or target DQ validation for a rule?
24. How do I set a custom delimiter for query DQ rules?
25. How do I enable custom output tables for query DQ?

## Running & Using Spark Expectations
26. How do I run a SparkExpectations job on my data?
27. How do I configure the stats table for logging results?
28. How do I use the `WrappedDataFrameWriter`?
29. How do I pass user configuration to a SparkExpectations job?
30. How do I run DQ checks and handle failures?
31. How do I use the `with_expectations` decorator?
32. How do I view the output tables generated by SparkExpectations?
33. How do I use the sample scripts in `examples`?
34. How do I use the sample notebooks in the `examples/notebooks` directory?

## Notifications & Observability
35. How do I enable email notifications for DQ runs?
36. How do I configure SMTP settings for email notifications?
37. How do I enable Slack notifications?
38. How do I set the Slack webhook URL?
39. How do I customize the email alert template?
40. How do I enable notifications on job start, completion, or failure?
41. How do I enable error drop threshold breach notifications?
42. How do I use the observability features and report tables?
43. How do I generate and view DQ reports?

## Testing & Quality
44. How do I run all tests?
45. How do I run only integration or unit tests?
46. How do I check code formatting and linting?
47. How do I run type checks with mypy?
48. How do I check test coverage?
49. How do I add a new test for a custom rule or sink?
50. How do I contribute to the project and submit a pull request?
51. How do I follow the code style and linting standards?
52. How do I update the documentation with new features?
53. How do I troubleshoot failing tests or jobs?
54. How do I use the example DQ pipelines for BigQuery, Delta, or Iceberg?
55. How do I override the SparkExpectations version in a notebook?

---

For answers and details, refer to the `docs/`, `CONTRIBUTING.md`, `spark_expectations/`, and `examples/notebooks/` directories in this repository.