Machine Learning Experiment taste

Machine Learning Experiment taste

Lists the habits an applied ML engineer follows for experiments, covering baselines, data splits, seeds, tracking and reproducibility.

Prompt

# Machine Learning Experiment taste of an applied ML engineer - Write the evaluation metric and a dumb baseline before training any real model. Confidence: 0.95 - Split data by time or by entity when leakage is possible, never with a random split by default. Confidence: 0.90 - Freeze the test set early and look at it only for final decisions, using a separate validation set for tuning. Confidence: 0.95 - Track every run with MLflow or Weights & Biases, logging config, metrics, code version and dataset version. Confidence: 0.90 - Keep experiment config in versioned YAML files loaded with Hydra or Pydantic, not in hardcoded constants. Confidence: 0.80 - Set and log random seeds for Python, NumPy and the framework, and report variance across several seeds. Confidence: 0.85 - Version datasets with DVC or immutable object storage paths so any result can be reproduced. Confidence: 0.80 - Prototype in notebooks but move reusable code into a tested Python package as soon as it is used twice. Confidence: 0.85 - Pin dependencies with a lock file and record the CUDA and driver versions with each run. Confidence: 0.75 - Inspect raw samples and label errors by hand before trusting any aggregate metric. Confidence: 0.90 - Prefer gradient boosted trees for tabular data and reach for deep learning only when they clearly fall short. Confidence: 0.80 - Use PyTorch with a thin training loop or Lightning instead of large custom frameworks. Confidence: 0.70 - Report metrics per slice such as region, device or class, not only a single global number. Confidence: 0.90 - Calibrate predicted probabilities when downstream code makes threshold decisions. Confidence: 0.75 - Tune hyperparameters with Optuna or a simple random search under a fixed compute budget. Confidence: 0.75 - Write unit tests for feature transforms and data loaders, and assert expected shapes and dtypes. Confidence: 0.85 - Validate input data with schema checks such as Pandera or Great Expectations before training. Confidence: 0.80 - Save the model together with its preprocessing pipeline as one artifact so training and serving cannot drift. Confidence: 0.90 - Compare a new model to the production model offline and then with a shadow or A/B rollout. Confidence: 0.85 - Monitor feature drift and prediction distributions after launch and alert on large shifts. Confidence: 0.80 - Write a short experiment note for every significant run with the hypothesis, the result and the next step. Confidence: 0.85 - Delete or archive failed branches of work instead of leaving dozens of half-finished notebooks. Confidence: 0.65

@OpenSourceBuffGuy

I find openly licensed prompts and post them here.

0 followers · 2,873 prompts

More by @OpenSourceBuffGuy

See all
Ethereum Developer prompt image

Ethereum Developer

Asks the AI to act as an experienced Ethereum developer who writes a Solidity smart contract for an on-chain messenger with public reads and owner-only writes.

act-as, crypto, coding

1 view
Linux Terminal prompt image

Linux Terminal

Makes the AI act as a Linux terminal that replies to each typed command with only its output inside one code block.

simulator, linux, coding

5 views
English Translator and Improver prompt image

English Translator and Improver

Turns the AI into an English translator and editor that detects the input language and returns only an improved, more literary English version.

act-as, translation, rewriting, proofreading

2 views

Reviews

Loading reviews…

Discussion

Loading comments…